Friday Feb 19, 2016

Oracle GoldenGate Adapter for MongoDB

Oracle GoldenGate can now integrate with Mongo DB using the GoldenGate Adapter for MongoDB. GoldenGate Adapter for MongoDB can run on the latest Java based Replicat available with Oracle GoldenGate for Big Data 12.2. It uses high performance native Java driver to MongoDB. It can handle automatic DDL replication and supports inserts, updates and deletes into MongoDB.

This project has converted to an open-source project for increased contribution from the Java community. The downloadable bundle contains source code, binary (jar) files, sample configuration files and instructions to use (readme.txt). You can download the bundle from URL https://java.net/projects/oracledi/downloads/directory/GoldenGate/MongoDB%20Adapter

Compatibility Matrix:

  • GoldenGate for Big Data: 12.2.0.1.0 (All platforms)
  • MongoDB Server : 2.4, 2.6, 3.0 3.2
  • MongoDB Java Driver: 3.2.2


Thursday Feb 18, 2016

Topology Configurations for Web Services in Oracle Data Integrator (ODI) 12.2.1

There are a couple of enhancements added in outbound Web Services invocation support in Oracle Data Integrator (ODI) 12.2.1. You can now configure the Web Service end point URLs in the Topology definition instead of hard coding it in the ODI Tools parameters. Logical Schema of such topology configuration can be then used directly in the OdiInvokeWebService tool parameters. Now ODI also allows you to invoke a WebService that is protected by OWSM policies and register those OWSM policies as part of Physical Schema definition.

To register the Web Services definition a new technology, “SOAP Web Service”, is added in the Topology where you create the Data Server for the Web Service and register its WSDL URL.

The Services, Ports, user credentials, OWSM policies for the endpoint service from the WSDL are configured as the Physical Schemas.

You can then create a Logical schema for it and bind it with Physical Schema through Context mapping. Such logical schema and context can be specified in the OdiInvokeWebService tool instead of hard coding Web Service URL and other parameters.

To learn more about the Topology Configurations for Web Services, there is an Oracle by Example tutorial available providing step by step instructions on using the feature: http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_12c/odi12c_newfeatures/ascii_files.html#section3s1

Additionally the new features introduced in Oracle Data Integrator 12.2.1 are also documented in our New Features white paper.

Tuesday Feb 16, 2016

More Tips on Oracle Data Integrator (ODI) for the Cloud from the A-Team

Check out this week’s post from Cecile Franchini of the A-Team, the title says it all: Tips for ODI in the Cloud: ODI On-Premise with DBCS.

If you are considering connecting to the Cloud and using Oracle DBCS – Oracle Database Cloud Service – ODI on-premise can help get the job done.

Friday Feb 12, 2016

Complex File Wizard is now available in ODI Studio 12.2.1

In the new Oracle Data Integrator 12.2.1 release the Native Format Builder wizard which is used by the Complex File technology is available directly within ODI Studio. Previously users had to use JDeveloper with the SOA Extensions to access it. The Native Format Builder wizard is used to parse a complex file structure and build a Native Schema (nXSD) description out of it, Oracle Data Integrator then uses this nXSD to reverse-engineer, read and write information from complex files.

You can find more information about the Native Format Builder and the Complex File technology in our documentation: http://docs.oracle.com/middleware/1221/odi/develop-connectivity-km/complex_files.htm#ODIKM1397

In addition you can find an Oracle By Example tutorial which will walk you through how to use this feature: https://apexapps.oracle.com/pls/apex/f?p=44785:24:103532390093139:::24:P24_CONTENT_ID,P24_PROD_SECTION_GRP_ID,P24_PREV_PAGE:12572,,24

Finally all the new features introduced in Oracle Data Integrator 12.2.1 are documented in our New Features white paper.


Thursday Feb 04, 2016

Oracle Big Data Preparation Cloud Service

In case you missed the recent executive webcast on Big Data Preparation Cloud Service, do not worry you can watch the replay here.

Senior Vice President of Development, Amit Zavery outlines the importance of cloud based data integration technologies and the emphasis that Oracle is placing on enabling not just the IT but also the business analysts with cutting edge requirements. He lays out the importance of streamlining the data preparation and data wrangling  process to not just be a necessary step towards analytics, but an enriching and empowering step towards self service data preparation.

Oracle Big Data Preparation Cloud Service

The webcast also featured Jeff Pollock, Vice President of Product Management, moderating the discussion and Vinay Kumar, Co-founder and CEO of Blue Cloud Innovations. Both of them also stressed the importance of a cloud based technology to bridge the IT-Business gap and how self service data wrangling when done right imparts great value, brings up rich recommendations and helps streamline and automate the data preparation pipeline.

About Big Data Preparation Cloud Service (BDP)

BDP is not just built for Big Data. It is built for business data of any size, from small departmental data to large enterprise data to massive internet data. The technology is based on Apache Spark ML and Natural Language Processing with semantic knowledge graphs on the Oracle Cloud. This means that it is more efficient in mapping relationships and recommendations, it offers a higher quality of of automation and better data enrichment.

To learn more about Oracle Big Data Preparation Cloud Service, visit us at our websites here and here.

 Meanwhile here is a selection of questions that were answered live during the webcast.

Questions

Answers

Please describe the scalability of this Big Data Preparation and Cloud Service.

Big Data Preparation runs native on Hadoop and Spark - it is able to handle large datasets leveraging the distributed processing the provisioned hadoop cluster. For more information about our service please follow these links: http://cloud.oracle.com/bigdatapreparation

https://www.oracle.com/middleware/data-integration/big-data-preparation-cloud-service/index.html

How does does the customer wait for their 'wrangled data" to be returned to them?

Depending on the use case, the customer may use the provided download results wizard or may leverage RESTful API's to automate the downloading of the prepared data from the Oracle Storage Cloud.

Can we use it with Data visualization service/BICS?

Yes. BDP produces a tab delimited output file that can be uploaded into either BICS or DVCS for analysis in Visual Analyzer. In the upcoming quarterly update, BDP will support direct integration with BICS - where BDP will be able to publish directly to BICS data store using the Datasync API.

I know you touched on this a few minutes ago, but I need a little more clarification on something. Did you state that the developer can possibly be eliminated in processing this data, or will it still be mandatory to have coding involved to acquire data?

The intent is not to eliminate IT entirely from the process but rather to empower business users and domain experts to be able to review and prepare potentially valuable datasets without getting IT involved too early in the process. Once a dataset has been validated by the business as being valuable and the process needs to be automated and operationalized, then IT would be involved and could leveraged the provided set of RESTful API's to process similar datasets as part of an automated process.

Could you compare it with Amazon AWS QuickSight Spice, Machine Learning tools?

They are two entirely different tools. AWS QuickSight is a Cloud Based BI tool that allows users to build data visualizations - BDP is purely a data preparation tool that prepares data for visualization and analytics.


Tuesday Feb 02, 2016

A-Team Article: Integrating Oracle Data Integrator (ODI) On-Premise with Cloud Services

Benjamin Perez-Goytia of the Data Integration Solutions A-Team provides a very matter of fact summary regarding Cloud Integration with Oracle Data Integrator (ODI). For all the details: Integrating Oracle Data Integrator (ODI) On-Premise with Cloud Services.

The blog post details how to integrate Oracle Data Integrator (ODI) on-premise with cloud services.  Cloud computing is a service or utility in high demand and enterprises find themselves with a mix of on-premise data sources and cloud services.  Ben describes how Oracle Data Integrator (ODI) on-premise can enable the integration of both on-promise data sources and cloud services.

Happy reading! For more A-Team reads on ODI, browse through the A-Team Chronicles.

Monday Feb 01, 2016

Oracle GoldenGate 12.2 Release

In keeping with our foundation principles, the focus for this release has been Usability, Manageability, Heterogeneity, and Performance.  

What’s New in Oracle GoldenGate 12.2 (Replication Simplification)

GoldenGate Platform Features –for Oracle, SQL Server, Sybase, Teradata, Informix, TimesTen, DB2 LUW, NonStop SQL/MX, PostgreSQL, and MySQL

Automatic Heartbeat Table
Record end-to-end replication lag and view details with a database view. Valid for heterogeneous configurations as well.

Metadata in Trails
No need to generate a definitions file or use SOURCEDEFS/ASSUMETARGETDEFS.

Parameter Insight
Lookup parameter definitions from GGSCI with INFO PARAM, get the list of running parameters for a process with SEND … GETPARAMINFO, and pre-validate parameter files with the new checkprm utility.

One Billion Trail Files
Trail file sequence length has increased to nine digits; a 1000x increase and effectively one billion trail files.

For the Oracle Database

  • DL replication between more than 2 Oracle databases
  • Seamless DML and DDL replication between tables of differing structures
  • Replicat integration with the Oracle Data Pump
  • Filters past table data loaded to target tables from expdp/impdp: DBOPTIONS ENABLE_INSTANTIATION_FILTERING
  • Integrated Capture support for invisible columns
  • Offload fetching to Active Data Guard
  • Oracle Clusterware command and control integration

For SQL Server

  • Remote ALO mode Capture
  • SQL Server Standard Edition Capture and Delivery
  • SQL Server 2014 Compressed Log backups
  • Column level character sets

For Teradata

  • Replicat can process truncates
  • Column level character sets
  • Capture from Teradata is no longer supported from this version onward

For DB2 LUW

  • Column level character sets
  • Temporal Tables

More information on Oracle GoldenGate 12.2

Downloads: http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

Documentation: http://docs.oracle.com/goldengate/c1221/gg-winux/index.html

Certification Matrix: http://www.oracle.com/technetwork/middleware/data-integration/goldengate12-2-x-cert-matrix-2769360.xls

Tuesday Jan 26, 2016

ODI 12c KMs are now available for Oracle Hyperion Planning and Essbase

Oracle Essbase and Oracle Hyperion Planning Knowledge Modules (KMs) are now available for Oracle Data Integrator 12.2.1. The patches containing the KMs are available for download on My Oracle Support. Patch# 20109114 contains Oracle Essbase KMs and Patch# 21903914 has Oracle Hyperion Planning KMs. The KMs support latest version (11.1.2.4) of these Hyperion Applications.  All the features available in their corresponding 11g KMs are available in these 12c KMs as well.

Monday Jan 25, 2016

Oracle Metadata Management (OMM) 12.2.1 is now available!

Metadata Management is essential to solving a wide variety of critical business and technical challenges which include how report figures are calculated, understanding the impact of changes to data upstream, providing reports in a business friendly way in the browser and providing reporting capabilities on the entire metadata of an enterprise for analysis and improvement. Oracle Metadata Management is built to solve all these pressing needs for customers in a lightweight browser-based interface.

With Oracle Metadata Management 12.2.1, you will find many new features and enhancements. To list just a few:

Oracle Metadata Management is now based upon 100% Java delivery and installation allowing support for Windows as well as variation Linux/Unix deployments. This provides all UI features and any bridges which are executable in a Java and Linux/Unix based environment, specifically 100% java based bridges including JDBC database bridges (Oracle, Teradata, DB2, SQL Server, etc.), big data bridges (Hadoop Hive, HCatalog), and other popular bridges such as CA ERwin xml, Informatica PowerCenter xml, Tableau BI, etc.

Oracle Metadata Management configuration architecture visualization is now based upon an HTML 5 based implementation which includes better scalability, performance and overall layout quality, and supports iPad and other tablets.

Oracle Metadata Management provides two primary user interfaces for access, analysis, management and administration. The Metadata Manager UI provides a more technical interface for those who are managers of the metadata or administrators of the repository. The Metadata Explorer UI is designed for optimized search, analysis and reporting, as well as more control over the context (configuration version) presented to given users.

Oracle Metadata Management is now offering detailed DI data flow lineage analysis on demand only (in real time), instead of pre-calculating. This enhancement means that harvesting times are reduced by about 80%!

Oracle Metadata Management has a great number of new or enhanced bridges covering RDBMS, Big Data, Data Modeling, Data Integration, and Business Intelligence.

More details on the bridges and all the New Features for this release can be found here.

You can download Oracle Metadata Management 12.2. 1 here!

Tuesday Dec 22, 2015

Oracle GoldenGate for Big Data 12.2.0.1 is Generally Available Now!

Much awaited Oracle GoldenGate for Big Data 12.2 is released today and it is available for download at OTN.

Let me give you a quick recap on Oracle GoldenGate for Big Data. Oracle GoldenGate for Big Data streams transactional data into big data systems in real-time, raising the quality and timeliness of business insights. Oracle GoldenGate for Big Data offers also provides a flexible and extensible solution to support all major big data systems.

Oracle GoldenGate for Big Data

  • Same trusted Oracle GoldenGate architecture used by 1000’s of customers
  • Data delivery to Big Data targets including NoSQL databases
  • Support for Polyglot, Lambda and Kappa architectures for streaming data

Key Benefits

  • Less invasive on source databases when compared to batch processing such as Sqoop or ETL processes
  • Simple ingestion for 1:1 data architecture for populating “raw data” zones
  • Real-time data delivery for streaming analytics/apps
  • Reliable, proven at scale with high performance


Architecture – GoldenGate for Big Data 12.2 versus 12.1

New Features in 12.2.0.1:



New Java based Replicat Process 

The advantages of using Java based Replicat process are the following:

    1. Improved performance with Java based adapters
    2. Declarative design and configurable mapping
    3. Transaction grouping based on Operation count & Message size
    4. Improved check pointing functionality
      E.g.: CHECKPOINTSECS 1 (default 10 seconds)

Dynamic Data Handling

You no longer require to define SOURCEDEFS. DDL changes are automatically replicated to target. For example, if a new column named “mycolumn“ is added on the source database, it will be automatically replicated to the target without stopping and reconfiguring Oracle GoldenGate.

Pluggable Formatters

Oracle GoldenGate for Big Data can write into any Big Data targets in various data formats such as delimited text or XML or JSON or Avro or custom format. This can save users cost and time for staging data in ETL operations.

Example: gg.handler.name.format= <value>
values supported are delimitedtext”, “xml”, “json”, “avro” or “avro_row”, “avro_op” or Custom Format. Extended class path needs to be included in the config file. <com.yourcompany.YourFormatter

Security Enhancement

Native Kerberos support is available in the 12.2.0.1 binaries.

Example of configuration:
gg.handler.gghdfs.authType=Kerberos
gg.handler.gghdfs.kerberosKeytabFile=/keytab/file/path
gg.handler.gghdfs.kerberosPrincipal=user/FQDN@MY.REALM

Declarative Design

Oracle GoldenGate for Big Data is able to provide mapping functionally between source table to target table and source field to target field for HDFS/Hive, HBase, Flume and Kafka. The metadata is also validated at Hive or using an Avro schema to ensure data correctness.

Example:
MAP GG.TCUSTOMER, TARGET GG.TCUSTMER2, COLMAP (USEDEFAULTS, "cust_code2"=cust_code,"city2"=city);

Kafka as target

Oracle GoldenGate for Big Data can write Logical change records data to a Kafka topic. Operations such as Insert, Update, Delete and Primary Key Update can be handled. It can handles native compression such as GZIP and Snappy in Kafka.

Example of defining Kafka Handler Properties:
gg.handlerlist=ggkafka
gg.handler.ggkafka.type=kafka
gg.handler.ggkafka.topicName=gg_kafka
gg.handler.ggkafka.mode=tx

Other Enhancements

  • Partition data by Hive Table and/or column. Partitioning into new file based on designated column values
    Example:
    • gg.handler.{name}.partitionByTable =true | false
    • gg.handler.{name}.partitioner.{fully qualified table name}={colname}
    • gg.handler.{name}.partitioner.{fully qualified table name}={colname1},{colname2}
    • gg.handler.<yourhandlername>.partitioner.dbo.TCUSTORD=region, rating
  • Configurable File Rolling Property for HDFS (file size, duration, inactivity timer, metadata change)
  • Configurable file output encoding into HDFS
  • Automatically create HBase table if it does not exist
  • Ability to treat primary key updates as a delete and then an insert in HBase
  • HBase row key generation
  • Treat Primary Key updates as delete and insert in Flume and HBase
  • New Time stamping functionality to include micro second precision as ISO-8601
  • Availability on additional OS platforms: Windows and Solaris
  • Certification for newer versions: Apache HDFS 2.7.x, Cloudera 5.4.x, Hortonworks 2.3, Kafka 0.8.2.0 and 0.8.2.1


For more details about new product features, you may refer to Oracle GoldenGate for Big Data 12.2.0.1 Release Notes and User Documentation.

For more information about Oracle GoldenGate for Big Data.

Feel free to reach out to me for your queries by posting in this blog or tweeting @thomasvengal

Happy Holidays ! 

Friday Dec 18, 2015

Oracle Management Pack for Oracle Data Integrator for Enterprise Manager 13.1 is out! Make the most of the value of your Oracle Data Integrator investments!

We’ve mentioned before that to help you make the most of Oracle Data Integrator, and to deliver a superior ownership experience in an effort to minimize systems management costs, Oracle’s Management Pack for Oracle Data Integrator is a great addition! As a quick refresher: this product leverages Oracle Enterprise Manager Cloud Control's advanced management capabilities to provide an integrated and top-down solution for your Oracle Data Integrator environments. Management Pack for Oracle Data Integrator supports both 11g (11.1.1.7.0 and higher) and 12c versions of Oracle Data Integrator (12.1.2, 12.1.3 and 12.2.1).

Management Pack for Oracle Data Integrator provides a consolidated view of your entire Oracle Data Integrator infrastructure. This enables users to monitor and manage all their components centrally from Oracle Enterprise Manager Cloud Control.


In review, important capabilities include:

Performance Monitoring and Management

Management Pack for Oracle Data Integrator streamlines the monitoring of the health, performance, and availability of each and all components of an Oracle Data Integrator environment – this includes Master and Work Repositories, Standalone and JEE agents, as well as source and target Data Servers.

Configuration Management

To ensure that the configurations of all critical Oracle Data Integrator components in production environments are consistent with staging or test environments, Management Pack for Oracle Data Integrator provides Configuration Snapshots to save working configurations into the Management Repository or into an external XML file. These snapshots can then be compared with the active configuration in staging or test environments, or against any historical snapshot. Configuration Comparison is the next logical step which thus allows you to ensure the consistency of configurations in your application environment. Going one step further, Configuration History keeps track of all configuration changes, which can help in diagnosing performance problems that may be related to configuration changes along the way.

Service Level Management

Management Pack for Oracle Data Integrator allows for a proactive method to monitor Oracle Data Integrator environments at all levels. Many out of the box metrics are automatically collected for each Oracle Data Integrator component (repositories, agents, sources, targets…) which then provides users the ability to set up alerts based on critical or warning thresholds. Notifications can be provided to users in various ways through emails or Simple Network Management Protocol (SNMP) options.

Historical Analysis and Reporting

While real-time metrics certainly come in handy, there is also something to be said for historical collection of data! Management Pack for Oracle Data Integrator stores collected metrics and configurations data in a central repository –allowing users to thus analyze the metrics in various incremental views that include; last 24 hours, last 7 days, last 31 days. In addition, customizable service and system dashboard functionality allows users to create reports on various services and systems for service level availability, usage, performance, and business indicators – all in an effort to facilitate strategic trend analysis and reporting.

In a nutshell, Oracle Management Pack for Oracle Data Integrator benefits include: reducing the costs associated with monitoring the overall Oracle Data Integrator infrastructure and providing visibility into complex Oracle Data Integrator deployments across the enterprise, minimizing troubleshooting and performance tuning efforts as well as improving the overall ability to efficiently monitor Service Level Agreement compliance using powerful alerting capabilities. (For additional detail and to review another blog on the subject, click here.)

Oracle Management Pack for Oracle Data Integrator is licensed separately. From an installation perspective it is packaged in the Fusion Middleware plug-ins for Enterprise Manager (version 12.1.0.4 and higher) and comes with the base Enterprise Manager 12c or 13c download. There is no separate installation file to download. More information can also be found here.

Try it out! Getting Started with Oracle Fusion Middleware Management Plug-in

Monday Dec 14, 2015

Oracle Enterprise Data Quality in the Leaders Quadrant

Oracle is a leader in the recently released Gartner Magic Quadrant for Data Quality Tools

Oracle Enterprise Data Quality (EDQ), Oracle's data quality offering is part of the Oracle Data Integration solutions portfolio.  EDQ offers extensive capabilities to profile, standardize and audit data for completeness and accuracy within organizations. The Oracle Enterprise Data Quality family of products helps organizations achieve maximum value from their business-critical applications by delivering fit-for-purpose data. These products also enable individuals and collaborative teams to quickly and easily identify and resolve any problems in underlying data. With Oracle Enterprise Data Quality products, customers can identify new opportunities, improve operational efficiency, and more efficiently comply with industry or governmental regulation.