X

Welcome to All Things Data Integration: Announcements, Insights, Best Practices, Tips & Tricks, and Trend Related...

Recent Posts

Oracle Cloud Infrastructure Data Integration

Understanding VCN Configuration for Oracle Cloud Infrastructure (OCI) Data Integration

Let's learn more about Oracle Cloud Infrastructure Data Integration. Today's blog will help you understand and teach you Virtual Cloud Network (VCN) configuration for Oracle Cloud Infrastructure Data Integration. Check out the previous blog written on Oracle Cloud Infrastructure Data Integration about Workspaces. Overview of Virtual Cloud Network (VCN) A virtual cloud network (VCN) is a customizable and private network in Oracle Cloud Infrastructure. Just like a traditional data center network, the VCN provides complete control over the network environment. This includes assigning own private IP address space, creating subnets, route tables, and configuring stateful firewalls. VCN resides within a single region but can cross multiple Availability Domains. Once users, groups, and compartments are created then start with VCN creation.  By default, there are two subnets in the VCN (Region Specific).  Private Subnet - Instances contain private IP addresses assigned to Virtual Network Interface Card (VNIC) Public Subnet - Contains both private and public IP addresses assigned to VNICs For more understanding of VCN can refer to - https://docs.cloud.oracle.com/en-us/iaas/Content/Network/Concepts/overview.htm Oracle Cloud Infrastructure Data Integration and Virtual Cloud Networks Now coming to the main topic "Understanding VCN with Oracle Cloud Infrastructure Data Integration". Oracle Cloud Infrastructure Data Integration is in the Oracle Tenancy which resides outside the user tenancy. For Data Integration to access the resources in the user tenancy and get the information related to VCN and subnets the following policy needs to be set at the compartment level/tenancy level i.e. policy set at default root compartment level. allow service dataintegration to use virtual-network-family in tenancy (or) allow service dataintegration to use virtual-network-family in compartment <vcn_compartment> Different Options when Creating a Workspace While creating workspaces there are two options provided i.e. Enable Private Network or using Public Network. Oracle Cloud Infrastructure Data Integration only supports regional subnets i.e. subnet across all Availability Domains. Regional subnets are used for high availability purposes.  While in the process of creating a Workspace using "Enable Private Network", Oracle Cloud Infrastructure Data Integration VCN gets extended with the user-selected VCN. When the option is not selected then Oracle Cloud Infrastructure services like Object Storage get accessed through Service Gateway defined at the tenancy level and the rest of the resources like Database are accessed through Public Internet. Let us consider multiple Scenarios to understand the Oracle Cloud Infrastructure Data Integration with VCN by selecting Private/Public subnet and accessing its resources. Before testing multiple scenarios following are pre-requisites created in the environment: Created VCN with the name "VCN_DI_CONCEPTS" in the respective compartment.   Created four subnets within the mentioned VCN. Oracle Cloud Infrastructure Data Integration only supports regional subnet. For more information on the regional subnets, refer to https://docs.cloud.oracle.com/en-us/iaas/Content/Network/Tasks/managingVCNs.htm Below is the list of resources created belonging to Subnet and Region while Testing   For Autonomous Data Warehouse (ADW) to be in private instance Network Security Group (NSG) needs to be defined. In NSG defined two ingress rule for PUBLIC_SUBNET_DI (10.0.2.0/24) and PRIVATE_SUBNET_DI (10.0.1.0/24)   For DB Systems in Private subnet, the following rules in the ROUTE table are included   For Service Gateway, select the option "All IAD Services in Oracle Services Network". To understand more about this option, refer https://docs.cloud.oracle.com/en-us/iaas/Content/Network/Tasks/servicegateway.htm   Scenario 1 - Accessing ADW, Object Storage, and Databases in the same Region using DI workspace in Private Subnet Oracle Cloud Infrastructure Data Integration workspace was created in PRIVATE_SUBNET_DI (10.0.1.0/24) Service Gateway used in the PRIVATE_SUBNET_DI     Scenario 2 - Accessing ADW, Object storage in different regions, and accessing Database Systems residing in a public subnet. To access ADW in different regions and DB Systems in public subnet a NAT Gateway is required.  Service Gateway is required for Object storage along with NAT Gateway for cross traffic. Route Rules screenshot(added NAT Gateway with the existing Service Gateway):     Scenario 3 - Accessing ADW, Object Storage, and Database in the same Region using DI workspace in Public Subnet OCI DI Workspace in Public Subnet "PUBLIC_SUBNET_DI" (10.0.2.0/24) Depending on the requirement if Oracle Cloud Infrastructure Data Integration Workspace has been assigned in a VCN and wants to connect resources residing in another VCN which might be in the same region or different region then Local or Remote peering is required accordingly. To understand more about Local or remote peering, refer https://docs.cloud.oracle.com/en-us/iaas/Content/Network/Tasks/VCNpeering.htm. If the resources are having Public IP then NAT or Service Gateway can be used accordingly. Scenario 4 - ADW, Object Storage, Database systems residing in the public subnet and all these resources are in different tenancy, different region, and different VCN To test this scenario we have created the following resources in the Mumbai region and different tenancy. The workspace of Oracle Cloud Infrastructure Data Integration is in Public Subnet (10.0.2.0/24). DI Workspace is created in the Ashburn region.   Scenario 5 - Connecting ADW, Databases and Object Storage using DI workspace with "Enable Private Network" Disabled While creating workspace if the option "Enable Private Network" is not selected   This non - enabling option means public connectivity option is selected where the Oracle Cloud Infrastructure Data Integration can access all the public services using Service Gateway and NAT Gateway from Oracle Tenancy. Here, Oracle Cloud Infrastructure Data Integration can't access private resources since for the workspace no VCN is assigned. In this example, Oracle Cloud Infrastructure Data Integration is enabled in the Ashburn region.     Scenario 6 - Connecting Oracle Cloud Infrastructure Data Integration with On-Premise DB There are two methods where Oracle Cloud Infrastructure Data Integration can connect to On-Premise DB IPSec VPN FastConnect Below are the details on how using FastConnect Oracle Cloud Infrastructure Data Integration can access the Database. To understand more about FastConnect, refer https://docs.cloud.oracle.com/en-us/iaas/Content/Network/Concepts/fastconnect.htm   Oracle Cloud Infrastructure Data Integration workspace should be in the same subnet where FastConnect is configured.   In the below example, VCN is created by Oracle as part of FastConnect with the name "####-iad.vcn"   Regional Public subnet is created within the VCN   Dynamic Route Gateway (DRG) is configured which is used as a virtual router that provides a path for private traffic (that is, traffic that uses private IPv4 addresses) between user VCN and networks outside the VCN's region. For more information on DRG, refer - https://docs.cloud.oracle.com/en-us/iaas/Content/Network/Tasks/managingDRGs.htm   DRG can be configured with IPSec or Oracle FastConnect   Within the DRG two virtual network have been configured using FastConnect   Route Rules defined in the VCN   OCI DI workspace created in the subnet   Under Data Asset Create and Test the connection   Summary - We can observe that Scenario 1 and Scenario 2 are the same irrespective of Subnet allocated to the workspace.  Since the secondary VNIC extended to the users VCN/tenancy is always Private. Oracle Cloud Infrastructure Data Integration Workspace is assigned to Public or Private Subnet Oracle Cloud Infrastructure Data Integration Workspace is not assigned any network - Disabled "Enable Private Network" Option We just recently announced the general availability of Oracle Cloud Infrastructure Data Integration. With a series of upcoming blogs, we look forward to introducing various concepts. This concludes our blog on how to use VCN in Oracle Cloud Infrastructure Data Integration. To learn more, check out some Oracle Cloud Infrastructure Data Integration Tutorials and the Oracle Cloud Infrastructure Data Integration Documentation.

Let's learn more about Oracle Cloud Infrastructure Data Integration. Today's blog will help you understand and teach you Virtual Cloud Network (VCN) configuration for Oracle Cloud Infrastructure Data...

Oracle Cloud Infrastructure Data Integration

Workspace in Oracle Cloud Infrastructure (OCI) Data Integration

Oracle Cloud Infrastructure Data Integration is a fully managed, serverless, native cloud service that helps you with common extract, load, and transform (ETL) tasks such as ingesting data from different sources, cleansing, transforming, and reshaping that data, and then efficiently loading it to a target system on Oracle Cloud Infrastructure. Before you get started, the administrator must satisfy connectivity requirements so that Oracle Cloud Infrastructure Data Integration can establish a connection to your data sources. The administrator then creates workspaces and gives you access to them. You use workspaces to stay organized and easily manage different data integration environments. The workspace is the preliminary component of Oracle Cloud Infrastructure Data Integration. The workspace acts as an environment provided where the user can work on multiple Projects, Publish/Run Tasks, and Define Data Assets. The administrator must define the policies for the users/groups to start with this data integration solution. Creating and Editing a Workspace: Pre-requisites - All the necessary compartments and VCN have been created for Data Integration activities.  To understand more about VCN for Oracle Cloud Infrastructure Data Integration, refer to https://docs.cloud.oracle.com/en-us/iaas/data-integration/using/preparing-for-connectivity.htm Create a group for users in charge of workspaces and then add users to the group. All the policies have been set up by the administrator so that the user can access the Oracle Cloud Infrastructure Data Integration. If the administrator wants to limit activities within the network, "inspect" permission for VCNs and subnets within the compartment has to be provided instead of "manage". Below is the list of policies required to access Oracle Cloud Infrastructure Data Integration Give permissions to the group to manage Oracle Cloud Infrastructure Data Integration allow group <group_name> to manage dis-workspaces in compartment <compartment_name> Give permission to the group to manage network resources for Workspaces allow group <group_name> to manage virtual-network-family in compartment <compartment_name> Give permission to the group to manage tag-namespaces and tags for Workspaces allow group <group_name> to manage tag-namespaces in compartment <compartment_name> Oracle Cloud Infrastructure Data Integration is located in Oracle Tenancy which is outside user Tenancy. Data Integration sends a request to user tenancy. In return, the user must give the requestor(DI) permission to use the virtual networks set up for integration. Without a policy to accept this request, data integration fails. These policies can be defined at the compartment level or the tenancy level i.e. at the root compartment level allow service dataintegration to use virtual-network-family in tenancy allow service dataintegration to inspect instances in tenancy   Select the Data Integration link from the main menu of Oracle Cloud Infrastructure   Select the corresponding compartment and click on "Create Workspace".   Provide necessary information i.e. Name for the Workspace, VCN details, and other information like DNS, Tag Names.   Click on create for creating the workspace in the corresponding compartment. You're returned to the Workspaces page. It may be a few minutes before your workspace is ready for you to access. After it's created, you can select a Workspace from the list.   You can see the status of a Workspace creation or startup using View Status. It is available while creating or starting a Workspace from Stopped Status   Workspace can be accessed in two ways as shown in the below picture   After accessing the workspace New Projects, Data Assets or Applications can be created through the main console of the Workspace   You can edit the Workspace details, such as a name or description. You can't make changes to the identifier, compartment, VCN, or subnet selections. To edit the tags applied to a Workspace, select Add Tags from the Workspace's Actions (three dots) menu. In the Console, you edit a workspace from the Workspaces page. Select Edit from a workspace's Actions (three dots) menu. Edit the fields you want to change, and then click Save Changes Terminating/Stopping a Workspace -  Only Workspaces with an Active status or in Stopped status can be terminated. When you terminate a workspace, all the associated objects and the resources are removed. Below is the list of resources: Projects Folders Data Flows Tasks Applications Task Runs Data Assets All executions within a Workspace must be stopped before you can terminate the Workspace. Any open tabs associated with the Workspace you're terminating are closed upon termination. Once terminated, a Workspace cannot be restored. Be sure to carefully review the Workspace and resources before you commit to a termination. To terminate the Workspace click on the workspace action(three dots) and then click on Terminate   We just recently announced the general availability of Oracle Cloud Infrastructure Data Integration. With a series of upcoming blogs, we look forward to introducing various concepts. This concludes our initial blog on how a Workspace can be created and used in Oracle Cloud Infrastructure Data Integration.  To learn more, check out some Oracle Cloud Infrastructure Data Integration Tutorials and the Oracle Cloud Infrastructure Data Integration Documentation.

Oracle Cloud Infrastructure Data Integration is a fully managed, serverless, native cloud service that helps you with common extract, load, and transform (ETL) tasks such as ingesting data from...

Data Integration

Oracle Named 2019 Gartner Peer Insights Customer Choice for Data Integration Tools

We are pleased to announce that Oracle has been recognized as a 2019 Gartner Peer Insights Customer Choice for Data Integration Tools.  This distinction is especially important to Oracle because it is based on the direct feedback from our customers.  Thank you all for your support! Oracle Data Integration provides an enterprise class, fully unified solution for building, deploying, and managing real-time data-centric architectures. It combines all the elements of data integration—real-time data movement, transformation, synchronization, data quality, data management, and data services—to ensure that information is timely, accurate, and consistent across complex systems.  By using Oracle Data Integration, customers can experience significant cost savings, and efficiency gains are critical to leverage in today's challenging global economic climate.  They are delivering real-time, enriched, and trusted data from disparate cloud and on-premises sources to enable insightful analytics.  “We are honored to receive Gartner Peer Insights Customers’ Choice designation for the Data Integration Tools market. We thank our customers for their support,” said Jeff Pollock, Vice President Product Management for Oracle. “Over the past 20 years Oracle Data Integration has evolved into an industry leading platform used by thousands of companies across every industry. Working together with our customers, Oracle is committed to driving the innovation necessary to solve the industry’s most challenging data integration issues.”  Find out more! Gartner Peer Insights is an enterprise IT product and service review platform that hosts more than 300,000 verified customer reviews across 430 defined markets. In markets where there is enough data, Gartner Peer Insights recognizes up to seven vendors that are the most highly rated by their customers through the Gartner Peer Insights Customers’ Choice distinction. According to Gartner, “The Gartner Peer Insights Customers’ Choice is a recognition of vendors in this market by verified end-user professionals.” To ensure fair evaluation, Gartner maintains rigorous criteria for recognizing vendors with a high customer satisfaction rate. We at Oracle are deeply proud to be honored as a 2019 Customers’ Choice for the Data Integration Tools Market. To learn more about this distinction, or to read the reviews written about our products by the IT professionals who use them, check out the Customers’ Choice Data Integration Tools for Oracle landing page on Gartner Peer Insights. Here are some excerpts of what Oracle Customers are saying: “Using GoldenGate, it is possible to carry out operations in high data volumes in a much faster and uninterrupted manner. It is also very easy to use. One of the most effective abilities is to manage transactional processing in complex and critical environments. It is very important that data, costs and ongoing transactions are regularly secured to bring the risk to near zero." Software Engineer, Finance Industry “ODI is a very good product. It is lightning fast (which really comes handy when we have to transform massive amount of data), It ability to support heterogeneous databases, big data, JMS, XML, and many other flavors.” Senior Manager - MIS & Middleware, Service Industry A big thank you to our wonderful customers who submitted reviews, and those customers who continue to use our product and services and help shape the future.     The GARTNER PEER INSIGHTS CUSTOMERS’ CHOICE badge is a trademark and service mark of Gartner, Inc., and/or its affiliates, and is used herein with permission. All rights reserved. Gartner Peer Insights Customers’ Choice constitute the subjective opinions of individual end-user reviews, ratings, and data applied against a documented methodology; they neither represent the views of, nor constitute an endorsement by, Gartner or its affiliates.

We are pleased to announce that Oracle has been recognized as a 2019 Gartner Peer Insights Customer Choice for Data Integration Tools.  This distinction is especially important to Oracle because it is...

GoldenGate Solutions and News

Oracle GoldenGate Plug-in for Oracle Enterprise Manager v13.2.3.0.0 is now available

We have released GoldenGate OEM Plug-in 13.2.3.0.0. The release's primary focus was to support the monitoring of Oracle GoldenGate 18.1, 19.1 Microservices (MA) Instances. In the earlier GoldenGate OEM Plug-in 13.2.2.0.0 release, we started supporting our first GoldenGate 12.3 Microservices Instance. In the new release, we have certified the latest of GoldenGate releases 18.1 and 19.1 Microservices and Classic. Along with the certification, we are supporting the new metrics for coordinated and parallel replicats. We have provided more services(Administration Service and Service Manager) and Deployments support in the plug-in. You may discover the new targets in the discovery module and promote the targets of your choice. Finally, we have certified the OEM 13.3 in the release.   Once you discover targets, you can select the processes(Extract, Replicat, etc) while promoting the targets. The selected processes(aka targets) and its parent process would get promoted automatically. For example, if you select the Extract process under Admin Server, the OEM PlugIn will promote the selected Extract process, Admin Server (which is Extract’s Parent), and Service Manager (which is Admin Server’s Parent). On a similar line, if you select the parent process, all the children will be selected by default and then you may choose to de-select the particular child.       Once you promote the processes or targets, you may notice the changes in Dashboard User Interface for Microservices processes. All the processes are shown in the tree structure. The Service Manager is the parent process, which shows one or many GoldenGate Deployments, all the extracts and Replicats are part of Admin Server. You may see all the services status on the screen. Along with the Microservices Instance, you may monitor the Classic Instance on the same dashboard. We have given each process(Target) a specific type name as per GoldenGate terminologies. It will be helpful when you want to know about what type of Extract or Replicats you are monitoring (Classic or Integrated Extract, Coordinated or Parallel Replicat).     When you click on the Service Manager on the dashboard, it will direct you to the below-mentioned page. The page shows all the Deployments and the details of its services, such as Port and Status. In the future, you should be able to search across a particular Deployment. The admin Server page will show all the extract and Replicats processes and its detailed metrics. When you click on individual Extract, Replicat you will be able to see the Metrics, Logs, and Configuration of the process.     For the Parallel and Coordinated replicat(PR/CR), you can see the accumulated metrics in the Parent process. The children process of the PR/CR is not visible on the screen. In the future, we will provide options to select the child process and then you would be able to monitor those children as well.   The GoldenGate OEM Plug-in has upgraded infrastructure to be compatible with the newer version of Enterprise Manager (EM) to 13.3.0.0.0. As mentioned earlier, we have certified the GoldenGate 18.1, 19.1 Classic and Microservices and added few metrics related to Parallel Replicat and Coordinated replicat.    Just to recap the communication between EM Agent and GoldenGate MA, and Classic Instances. You would not require to setup GoldenGate jAgent(Monitor Agent) to communicate with GoldenGate OEM Plug-in for GoldenGate Microservices Instances. The GoldenGate MA architecture provides the RESTful APIs to monitor and manage the GoldenGate MA Instances. The GoldenGate OEM Plug-in uses these RESTful APIs to communicate with GoldenGate MA Instances. For your GoldenGate Classic Instances, you would still need to setup GoldenGate jAgent 12.2.1.2.0+ for the communication purposes. The latest Monitor Agent was released on May, 19 (12.2.1.2.190530).   You can get more details of the release from the documentation.    We are working to get more features around monitoring the GoldenGate Microservices and Classic architecture in future releases. Please stay tuned for further updates.  

We have released GoldenGate OEM Plug-in 13.2.3.0.0. The release's primary focus was to support the monitoring of Oracle GoldenGate 18.1, 19.1 Microservices (MA) Instances. In the earlier GoldenGate OEM...

Release Announcement for Oracle GoldenGate 19.1

This post was authored by Bobby Curtis, Director of Product Management, Oracle What’s New in Oracle GoldenGate 19.1 To succeed in today’s competitive environment, you need real-time information. This requires a platform that can unite information from disparate systems across your enterprise without compromising availability and performance. Oracle GoldenGate 19c is a high-performance software application for real-time transactional change data capture, transformation, and delivery, offering bidirectional data replication. The application enables you to ensure that your critical systems are operational 24/7, and the associated data is distributed across the enterprise to optimize decision-making. GoldenGate 19.1 Platform New Features For the Oracle Database ➢ Oracle Database 19c Support Capture and Delivery support for Oracle Database 19c, cloud and on-premises. ➢ Centralized Key Management Service Use Oracle Key Vault to centralize and manage encryption keys for the replication environment. ➢ Target-Initiated Paths Distribution paths enabled from the Receiver Service to pull trail files. ➢ New REST API Endpoints Retrieve active transactions and current system change number (SCN) details using REST API endpoints. ➢ New Heartbeat Table Command The UPGRADE HEARTBEATTABLE command upgrades the Heartbeat table from prior versions of Oracle GoldenGate to the 19.1 version. ➢ Cross-Endian Support for Remote Integrated Extract Automatically enabled when the server where the Integrated Extract is running is different from the server where the Oracle Database is running. For MySQL ➢ MySQL 8.0 Support for Capture and Delivery Capture and Delivery support for MySQL 8.0 has been added. ➢ MySQL SSL Connection Support Extract and Replicat can now connect to a MySQL database via SSL. For DB2 for i ➢ Enhanced TIMESTAMP Support Supports all valid TIMESTAMP precisions ➢ New Datatype Support Support for DECFLOAT datatype. ➢ New DBOPTIONS USEDATABASEECODING Parameter Allows Extract to store all text data in the trail file in native character encoding. ➢ Improved Extract Performance Enhanced throughput while reducing overall processing. ➢ Security Improvements Availability of AES Encryption. Credential Store, and Oracle Wallet. ➢ Long Running Transaction (LRT) Support Support for LRT features showtrans, skiptrans, forcetrans. For DB2 z/OS ➢ Enhanced TIMESTAMP Support Supports all valid TIMESTAMP precisions ➢ Online Schema Change Support Support for online TABLE CREATE, DROP and ADD, ALTER, DROP COLUMN commands. ➢ Long Running Transaction (LRT) Support Support for LRT features showtrans, skiptrans, forcetrans. For DB2 LUW ➢ Enhanced TIMESTAMP Support Supports all valid TIMESTAMP precisions ➢ New Datatype Support Support for DECFLOAT datatype. ➢ Long Running Transaction (LRT) Support Support for LRT features showtrans, skiptrans, forcetrans. Other Information ➢ In the initial release, OGG 19.1.0.0.0, Linux builds will be available for most Database/OS combinations that are supported, followed by tiered releases for other supported platforms. ➢ GoldenGate for SQL Server will be released for both Windows and Linux soon, in a 19.1.x release. Docs, Downloads, and Certification: • Documentation is available at: https://docs.oracle.com/en/middleware/goldengate/core/19.1/index.html • Downloads are available through OTN at: https://www.oracle.com/middleware/technologies/goldengate.html • Certification Matrix (19.1 Cert Matrix to be posted soon): https://www.oracle.com/technetwork/middleware/ias/downloads/fusion-certification-100350.html Join us in upcoming events: • Stay up to date by visiting our Data Integration Blog for up to date news and articles. • Save the Date! Oracle OpenWorld is September 16th through the 19th. Don’t hesitate to contact us for any special topics that you might like to discuss. 

This post was authored by Bobby Curtis, Director of Product Management, Oracle What’s New in Oracle GoldenGate 19.1 To succeed in today’s competitive environment, you need real-time information. This...

GoldenGate Solutions and News

Zero Down Time (ZDT) Patching for Oracle GoldenGate

  This document explains how to apply a patch or upgrade an OGG environment without taking any downtime.  This assumes that OGG is already up and running and that the user is already very familiar with how OGG works, and the actual upgrade process.  Like any mission critical, 24x7 environment, this expectation is that the user takes the necessary precautions to test this process prior to implementing it in production, and is aware of any differences between versions.  All of these items are covered in other documents.  Terminology “New” – This refers to the new OGG installation.   This “new” environment is where everything will be running once you have completed the procedure. “Old” – This refers to the old OGG installation.  This “old” environment is the existing OGG installation that you want to upgrade.  After the process is completed, you will be removing this installation. Patching OGG Homes where there are Extract(s) running. Install the new OGG version into a new directory. This location will be referred to as the “new” OGG installation. In the new installation Apply any necessary patches to bring the releases to the most recent bundle patch, and then apply any required one-off patches on top of that. Create new Extract process(es) with different names than the old OGG environment. Create a new set of trail files (different names than the old OGG installation. Copy the parameter files from the old installation into the new one.  Modify them to account for new directories, names, and address any deprecated / modified parameters. On the target Create a new Replicat to receive data from the new OGG installation. In the new Installation Start the Extract Start the Extract pump (if necessary) In the old installation Wait.   How long to wait for?  It depends.  When you started the new Extract in step 4a, it will not process any transactions that were open when it was started.  You will want to wait until any open transactions during that time are closed.  SEND EXTRACT … SHOWTRANS may help in this case. Stop the Extract On the target If the old Replicat is not using a checkpoint table ,add one for it. Once the Replicat from the old installation is at EOF (SEND REPLICAT … GETLAG) stop the old replicat. Start the new replicat using START REPICAT … AFTERCSN [scn].  Where the [scn] is the log_cmplt_csn column from the checkpoint table for the old replicat. This will tell the new replicat to pick up right where the old replicat left off. In the old installation Stop the extract pump (optional) Clean up the old installation and remove it.   Patching OGG Homes where there are Replicat(s) running. Install the new OGG version into a new directory. This location will be referred to as the “new” OGG installation. In the new installation Apply any necessary patches to bring the releases to the most recent bundle patch, and then apply any required one-off patches on top of that. Create new Replicat process(es) with different names than the old OGG environment.  The new replicat will read from the existing trail files. Copy the parameter files from the old installation into the new one.  Modify them to account for new directories, names, and address any deprecated / modified parameters. In the Old installation. If the old Replicat is not using a checkpoint table ,add one for it. Stop the Replicat when it is at EOF (SEND REPLICAT … GETLAG) In the New Installation Start the new replicat using START REPICAT … AFTERCSN [scn].  Where the [scn] is the log_cmplt_csn column from the checkpoint table for the old replicat. This will tell the new replicat to pick up right where the old replicat left off. In the old installation Clean up the old replicat and remove it.  

  This document explains how to apply a patch or upgrade an OGG environment without taking any downtime.  This assumes that OGG is already up and running and that the user is already very familiar with...

Demystifying Oracle Cloud Infrastructure

Oracle has a longstanding reputation for providing technologies that empower enterprises to solve demanding business problems. Oracle has built a cloud infrastructure platform that delivers unmatched reliability, scalability, and performance for mission-critical databases, applications, and workloads. Oracle Cloud Infrastructure is the first cloud built specifically for the enterprise. With the latest high-end components, support for open standards and multi-cloud strategies, and an unwavering commitment to protecting sensitive business data, Oracle Cloud Infrastructure is perfectly suited to meet the needs—and exceed the expectations—of today's enterprise IT teams. Oracle Cloud Infrastructure represents a fundamentally new public cloud architecture and serves as the foundational layer for Oracle Cloud. The infrastructure is designed to provide the performance predictability, core-to-edge security, and governance required for enterprise workloads. Oracle supports traditional, mission-critical, and performance-intensive workloads typically found in on-premises environments, including artificial intelligence (AI), machine learning (ML), and high-performance computing (HPC), as well as cloud-native applications. Oracle Cloud Infrastructure combines the benefits of public cloud (on-demand, self-service, scalability, pay-for-use) with those benefits usually associated with on-premises environments (governance, predictability, control) into a single offering. Here is a good example of how Alliance Data Saves $1 Million Annually Running Critical Applications on Oracle Cloud Infrastructure Learn more about Oracle Cloud Infrastructure here.

Oracle has a longstanding reputation for providing technologies that empower enterprises to solve demanding business problems. Oracle has built a cloud infrastructure platform that delivers unmatched...

Enabling Analytics with Oracle data integration and Oracle Analytics Cloud on Oracle Autonomous Database

Enabling global analytics is one of the most common use cases among customers who build and maintain a data store. In this post, we shall identify the critical components that are required for an end-to-end analytics solution, the characteristics of a great analytics solution, and how Oracle Analytics Cloud, Oracle data integration, and Oracle Autonomous Database, together combine to provide a platform for great analytics. Any chosen analytics solution should bring together and balance the requirements of two major stakeholders, those in the Information Technology (IT) departments and those in the line-of-business functions. Fig 1: IT and Business dictates priorities that need to be balanced in an analytics solution Achieving this balance between the scalability requirements of IT and the user experience focus of a visual tool is critical to the success of any visualization solution. Oracle Data Integration -  The IT Component Oracle data integration solutions help solve key requirements for a successful IT deployment of an analytics solution. Oracle data integration Provides the latest data, both in real-time and bulk, from various sources to be delivered into the data warehouse that is built on top of Oracle Autonomous Database to power analytics, Helps govern data and provide transparency to the data that underpins the analytics visualizations, for easy lineage and impact analysis, thus increasing trust in the data, and Enables true global analytics, by making data available both on-premises and on the cloud. Oracle Analytics Cloud - The Business Component Oracle Analytics Cloud provides the features and benefits that satisfy the requirements of a business user. Oracle Analytics Cloud Provides powerful data flows and enrichment features to enable sharable and traceable business data transformations, Avoids Excel clutter and empower analysts to enhance data with no coding skills required, and Enables augmented data enrichment, through Machine Learning driven enrichment and data transformations. Oracle Autonomous Database - The Platform  Oracle Autonomous Database forms the third component of this analytics solution, along with Oracle data integration and Oracle Analytics Cloud. Oracle Database Provides a robust self- driving, self-securing, and self-repairing data store, providing autonomous datawarehousing capabilities, Watch this video to understand how these three components come together to provide end to end analytics on an Oracle platform. Fig 2: Oracle data integration video  Oracle Data integration, along with Oracle Autonomous Data Warehouse and Oracle Analytics accelerates speed to insight and innovation while enabling fast access to a sophisticated set of analytics and accelerates data preparation and enrichment. Watch this webcast to learn more about how to focus on growing your business and drive innovation with an end-to-end analytics solution.

Enabling global analytics is one of the most common use cases among customers who build and maintain a data store. In this post, we shall identify the critical components that are required for...

Data Integration

Loading Data Into Oracle Autonomous Data Warehouse Cloud with Oracle data integration

Oracle offers the world’s first autonomous database. Oracle also offers tools that helps customers get data into the autonomous database.  In this blog, we will go through what is an Autonomous Database and the capabilities that Oracle data integration provides that helps adopt the Autonomous Data Warehouse Cloud Service (ADWCS). What is an Autonomous database? An autonomous database is a cloud database that uses machine learning to eliminate the human labor associated with database tuning, security, backups, updates, and other routine management tasks traditionally performed by database administrators (DBAs). Autonomous database technology requires that enterprise databases be stored in the cloud, using a cloud service. Being autonomous in the cloud allows the organization to leverage cloud resources to more effectively deploy databases, manage database workloads, and secure the database. A database cloud service makes database capabilities available online, when and where those capabilities are needed. Watch Senior Vice President Juan Loaiza introduce the Oracle Autonomous Database for a deeper insight into the technology. What is Oracle data integration? Oracle’s data integration encompasses a portfolio of cloud-based and on-premises solutions and services that helps with moving, enriching, and governing, data. Oracle data integration has the following capabilities that make it the logical choice when looking to migrate and move data to Oracle Cloud. Oracle data integration Has integrated APIs that allow easy access to Oracle’s underlying tables without affecting source system performance for real-time data access through change data capture, Can automate repeated data delivery into Oracle Datawarehouse Cloud Service by easily surfacing ADWCS as a target system, Brings together real-time data replication, data streaming, bulk data movement, and data governance into a cohesive set of products that are seamlessly integrated for performance. Watch this video to get a quick glimpse of our latest product and how it functions with Oracle Data Warehouse Cloud and Oracle Analytics cloud. Moving Data Into Oracle Data Warehouse Cloud Service Oracle data integration solutions bring together some key technological and user benefits for customers. Managed by Oracle – Engineered and built by teams that have a shared vision, the different solutions and technologies incorporate the best of scientific advances, as well as, seamless integration between the solutions. Unified Data Integration – Provides a single-pane-of-glass to control the various components of data integration like bulk data movement, real-time data, data quality, and data governance. Simplify Complex Integration Tasks – Groups together functions that build up to a business or technology pattern, so that often repeated scenarios can be executed with efficiency. Flexible Universal Credit Pricing – Oracle’s pricing tracks usage, and can be applied across technologies, allowing customers access to all participating Oracle cloud services, freeing customers from procurement woes and providing customers with a truly agile and nimble set of solutions. Here are some scenarios that Oracle data integration helps to solve. Extraction & Transformation - Execute bulk data movement, transformation, and load, scenarios, Data Replication – Change data capture helps replicate data into Oracle Autonomous DataWarehouse and Kafka, for data migration and high availability architectures, Data Lake Builder - Create a comprehensive, fully governed, repeatable data pipeline to your big data lakes, Data Preparation - Ingest and harvest metadata for better data transparency and audits, and Synchronize Data - Seamlessly synchronize two databases together. Fig1: A sample architecture of moving data from source to analytics For a deeper understanding of moving data into Oracle Autonomous Data Warehousing Cloud, watch the below webcast.

Oracle offers the world’s first autonomous database. Oracle also offers tools that helps customers get data into the autonomous database.  In this blog, we will go through what is an Autonomous...

GoldenGate Solutions and News

Oracle GoldenGate for SQL Server supports SQL Server 2017 and Delivery to Microsoft Azure SQL Database

The Oracle GoldenGate Product Management team is pleased to announce that Oracle GoldenGate 12.3 for SQL Server has added new functionality to support Capture and Delivery from/to SQL Server 2017 Enterprise Edition and has added certification to deliver to Microsoft’s Azure SQL Database. SQL Server 2017 Using patch release 12.3.0.1.181228 of Oracle GoldenGate for SQL Server (CDC Extract), which is available on support.oracle.com, under Patches & Updates, customers now have the ability to both capture from and deliver to SQL Server 2017 Enterprise Edition.  Azure SQL Database Also, using the same patch release as for SQL Server 2017 support, remote delivery to Azure SQL Database is now supported.  You can install the Oracle GoldenGate patch on a supported Windows server (see the Certification Matrix) and configure a remote Replicat to deliver data to your Azure SQL Database. Documentation For more information, please review the Oracle GoldenGate documentation as well as a quick start tutorial, which is available here: https://apexapps.oracle.com/pls/apex/f?p=44785:24:111923811479624::NO:24:P24_CONTENT_ID,P24_PREV_PAGE:21869,1

The Oracle GoldenGate Product Management team is pleased to announce that Oracle GoldenGate 12.3 for SQL Server has added new functionality to support Capture and Delivery from/to SQL Server 2017...

Data Integration

Integration: Heart of the Digital Economy Podcast Series – Moving Data to the Cloud and Autonomous Data Warehouse

Authored by Steve Quan, Principal Product Marketing Director, Oracle Digital transformation is inevitable if want to thrive in today’s economy.  We've heard about how application and data integration play a central role in business transformations.  Since data has become a valuable commodity, integration plays a critical role in sharing data with applications in hybrid cloud environments or populating data lakes for analytics.  In these two podcasts you can learn how easy it is to seamlessly integrate data for these use cases. Successful digital businesses rely on data warehouses for contextual information to identify customer intent and remain one-step ahead of competition.   With growing data volumes, you need to easily acquire and prepare data in the right format for business intelligence and analysis.  Listen to Integrating Data for Oracle and Autonomous Data Warehouse  and learn how easy it is to move and keep your data synchronized. Data is also moving to hybrid cloud environments so you can use data on-premises and in the cloud; enabling your organizations to be more agile and react quickly to changes.  Moving data to the cloud is not just copying initial blocks of data, you need to move the data and keep the data synchronized. Listen to Moving Data into the Cloud and learn how Oracle Data Integration makes this easier. Learn more about Oracle’s Application Integration Solution here. Learn more about Oracle’s Data Integration Solution here. Dive into Oracle Cloud with a free trial available here.   Oracle Cloud Café Podcast Channel - check out the Oracle Cloud Café, where you can listen to conversations with Oracle Cloud customers, partners, thought leaders and experts to get the latest information about cloud transformation and what the cloud means for your business.

Authored by Steve Quan, Principal Product Marketing Director, Oracle Digital transformation is inevitable if want to thrive in today’s economy.  We've heard about how application and data integration...

Data Integration

DATA REPLICATION TO AWS KINESIS DATA STREAM USING ORACLE GOLDENGATE

Contributed by: Shrinidhi Kulkarni, Staff Solutions Engineer, Oracle Use case: Replication of data trails present on AWS AMI Linux instance into Kinesis Data Stream (AWS Cloud) using Oracle GoldenGate for Big Data. Architecture: GoldenGate For Big Data: Oracle GoldenGate 12.3.2.1 AWS EC2 Instance: AMI Linux Amazon Kinesis  Highlights: How to configure GoldenGate for Big Data(12.3.2.1) How to configure GoldenGate Big Data Target handlers How to create AWS Kinesis Data Stream Connecting To Your Linux Instance from Windows Using PUTTY Please refer to the following link & the instructions in it that explain how to connect to your instance using PUTTY. And also on how to Transfer files to your instance using WinSCP.     https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html Download the GoldenGate for Big Data Binaries, Java (JDK or JRE) version 1.8 & Amazon Kinesis Java SDK Download and install GoldenGate for Big Data 12.3.2.1.1, Here is the link: http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html The Oracle GoldenGate for Big Data is certified for Java 1.8. Before installing and running Oracle GoldenGate 12.3.2.1.1, you must install Java (JDK or JRE) version 1.8 or later. Either the Java Runtime Environment (JRE) or the full Java Development Kit (which includes the JRE) may be used. The Oracle GoldenGate Kinesis Streams Handler uses the AWS Kinesis Java SDK to push data to Amazon Kinesis. The Kinesis Steams Handler was designed and tested with the latest AWS Kinesis Java SDK version 1.11.429 and for creating streams/ shards. https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/using-kinesis handler.htm#GADBD-GUID-3DE02CFE-8A38-4407-86DF-81437D0CC4E2 Create a Kinesis data stream(not included under Free-tier)on your AWS Instance, Follow the link for reference- https://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-create-stream.html It is strongly recommended that you do not use the AWS account root user or ec2-user for your everyday tasks, even the administrative ones. You need to create a new user with access key & secret_key for AWS, use the following link as reference to do the same :            https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html Attach the following policies to the newly created user to allow access and GET/Put Operations on Kinesis data stream: AWSLambdaKinesisExecutionRole-Predefined Policy in AWS You need to attach the following inline policy as json:  "Version": "2012-10-17",  "Statement": [    {    "Effect": "Allow",      "Action": "kinesis:*",      "Resource": [        "arn:aws:kinesis:<your-aws-region>:<aws-account-id>:stream/<kinesis-stream-name>"      ]    } Unzip the GoldenGate for big data (12.3.2.1) zip file : After you Unzip the Downloaded GoldenGate for Big Data Binary, the directory structure looks like this: Now extract the GoldenGate 12.3.2.1.1 .tar file using “tar -xvf” command. After the “tar –xvf” operation finishes, the following Big-Data target handlers are extracted: You can have a look on the directory structure( files extracted) and then go to “AdapterExamples” directory to make sure kinesis streams handler is extracted:           The Kinesis_Streams directory under big-data contains Kinesis Replicat parameter file(kinesis.prm) and kinesis properties file (kinesis.props). Before you log into GoldenGate instance using GGSCI, set the JAVA_HOME & LD_LIBRARY_PATH to the JAVA 1.8 directory otherwise it would show up an error as following: Export the JAVA_HOME & LD_LIBRARY_PATH as shown below:              export JAVA_HOME=<path-to-your-Java-1.8>/jre1.8.0_181              export LD_LIBRARY_PATH=<path-to-your-Java-1.8>/lib/amd64/server:$JAVA_HOME/lib Once you’re done, log into GoldenGate Instance using ./ggsci command and issue create subdir command to create the GoldenGate specific directories: Configure the Manager parameter file and add an open PORT to it: Example: edit param mgr          PORT 1080 Traverse back to GoldenGate Directory, execute ./ggsci and Add replicat in the GoldenGate instance using the following command:      add replicat kinesis, exttrail AdapterExamples/trail/tr [NOTE: A demo trail is already present at the location: AdapterExamples/trail/tr] Copy the parameter file of the replicat (mentioned above) to ./dirprm directory of the Goldengate Instance. Copy the properties file (kinesis.props) to dirprm folder after making the desired changes. Replicat Param File & kinesis properties file: REPLICAT kinesis -- Trail file for this example is located in "AdapterExamples/trail" directory -- Command to add REPLICAT -- add replicat kinesis, exttrail AdapterExamples/trail/tr TARGETDB LIBFILE libggjava.so SET property=dirprm/kinesis.props REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 1 MAP QASOURCE.*, TARGET QASOURCE.*; Kinesis Properties File(kinesis.props): gg.handlerlist=kinesis gg.handler.kinesis.type=kinesis_streams gg.handler.kinesis.mode=op gg.handler.kinesis.format=json gg.handler.kinesis.region=<your-aws-region> #The following resolves the Kinesis stream name as the short table name gg.handler.kinesis.streamMappingTemplate=<Kinesis-stream-name> #The following resolves the Kinesis partition key as the concatenated primary keys gg.handler.kinesis.partitionMappingTemplate=QASOURCE #QASOURCE is the schema name used in the sample trail file gg.handler.kinesis.deferFlushAtTxCommit=true gg.handler.kinesis.deferFlushOpCount=1000 gg.handler.kinesis.formatPerOp=true #gg.handler.kinesis.proxyServer=www-proxy-hqdc.us.oracle.com #gg.handler.kinesis.proxyPort=80 goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=DEBUG gg.report.time=30sec gg.classpath=<path-to-your-aws-java-sdk>/aws-java-sdk-1.11.429/lib/*:<path-to-your-aws-java-sdk>/aws-java-sdk-1.11.429/third-party/lib/*   ##Configured with access id and secret key configured elsewhere javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar   ##Configured with access id and secret key configured here javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar -Daws.accessKeyId=<access-key-of-new-created-user> -Daws.secretKey=<secret-ke-new-created-user> Make sure you edit the classpath, accessKeyId & Secret Key (of newly-created-user) correctly. After making all the necessary changes you can start the kinesis replicat, which would replicate the trail data to kinesis Data stream. Crosscheck for kinesis replicat’s status, RBA and stats. Once you get the stats, you can view the kinesis.log from. /dirrpt directory which gives information about data sent to kinesis data stream and operations performed.          You can also monitor the data that has been pushed into Kinesis data stream through AWS CloudWatch. Amazon Kinesis Data Streams and Amazon CloudWatch are integrated so that you can collect, view, and analyze CloudWatch metrics for your Kinesis data streams. For example, to track shard usage, you can monitor the following metrics: IncomingRecords: The number of records successfully put to the Kinesis stream over the specified time period. IncomingBytes: The number of bytes successfully put to the Kinesis stream over the specified time period. PutRecord.Bytes: The number of bytes put to the Kinesis stream using thePutRecord operation over the specified time period.

Contributed by: Shrinidhi Kulkarni, Staff Solutions Engineer, Oracle Use case: Replication of data trails present on AWS AMI Linux instance into Kinesis Data Stream (AWS Cloud) using Oracle GoldenGate...

Data Integration

Data Integration Platform Cloud (DIPC) 18.4.3 is Now Available

Data Integration Platform Cloud (DIPC) 18.4.3 is now available! Do you know what DIPC is?  If not, check out this short 2 minute video! Data Integration Platform Cloud (DIPC) is a re-imagination of how various best of breed data integration solutions can come together and work seamlessly, finding synergies in their features and elevating smaller piecemeal tasks and projects into a solution based approach. For example, DIPC introduces the concept of “elevated tasks” and “atomic tasks”. The latter, atomic tasks, are equivalent to point tasks that are used to accomplish smaller data requirements and logic, while the former, elevated tasks, consists of end goal oriented (e.g. building a data lake, or prepping data) groupings that bring together often encountered technological requirements into simple and logical task groupings. Let’s explore some of the new features for DIPC 18.4.3: A major enhancement we made in this release is the added support for Autonomous Data Warehouse (ADW), Oracle’s easy-to-use, fully autonomous database that delivers fast query performance. You can now create a connection to ADW and harvest metadata that can be used in our elevated tasks. In a recent blog article we explored the Data Lake Builder task.  This task helps with data lake automation, enabling an intuitive instantiation and copy of data into a data lake, in an effort to help reduce some of the existing data engineer/ data scientist friction.  You can quickly create a comprehensive, end-to-end repeatable data pipeline to your data lake.  The Add Data to Data Lake task now supports Autonomous Data Warehouse as a target and you can also ingest from Amazon S3.  Additionally, task execution is supported through the remote agent. The Replicate Data task includes advanced Kafka support with Avro and Sub types.  The user experience has been enhanced to support many varied replication patterns in the future.  You also have the option to encrypt data within the task. The ODI Execution task adds support for Autonomous Data Warehouse (ADW) and Oracle Object Storage empowering users to bulk load into ADW and run ETL/ELT workloads to transform their data. You’ll also find that DIPC scheduling is offered, allowing you to create scheduling policies to run jobs.  Additionally, heterogeneous support has been expanded, with GoldenGate for SQL Server now available through the DIPC agent. Learn more by checking out this documentation page for bits on how to create and runs tasks.

Data Integration Platform Cloud (DIPC) 18.4.3 is now available! Do you know what DIPC is?  If not, check out this short 2 minute video! Data Integration Platform Cloud (DIPC) is a re-imagination of...

Oracle Open World 2018 - A Recap

The Annual Oracle Tech Bonanza The first time I attended Oracle’s Open World, in 2013, was when I truly understood the scale of innovation and expertise that Oracle brings to its customers. Over the years, each year, I have only been more amazed at the breadth of technologies, various success stories, and incredible innovation, that Oracle and our customers combine to bring change to the way businesses operate. This year was no different. Oracle Open World 2018 had some of the most relevant and thought-provoking ideas, cutting-edge product showcases, and real-world use cases, on display. Here is a statistic that might throw a light on the scale of the event. “This week Oracle OpenWorld 2018 has hosted more than 60,000 customers and partners from 175 countries and 19 million virtual attendees. Oracle OpenWorld is the world’s most innovative cloud technology conference and is slated to contribute $195 million in positive economic impact to the City of San Francisco in 2018.” For a quick overview of the entire conference, you can read the full press release here.                           Key Notes Starting off the conference every day were keynotes that set the tone for the technology fest every day. Larry Ellison, Executive Chairman and Chief Technology Officer, kicked off the conference diving deep into two mainstays, among others, of Oracle’s focus, namely, Oracle Fusion Cloud Applications, and, Oracle’s Autonomous Database services. With an increased focus on intelligent platforms and a rich cloud ecosystem, Oracle’s Cloud is a critical component that glues together the dizzying array of services and solutions that Oracle offers. Some of the other keynote speakers, in no particular order, included, Mark Hurd, Chief Executive Officer, Steve Miranda, Executive Vice President, and Judith Sim, Chief Marketing Officer. In case you missed it, or want to revisit these sessions, you can watch them all here. My personal favorite was the session on The Role of Security and Privacy in a Globalized Society. To listen to the heads of highly performant teams discuss real-world problems using our technologies drives home the importance of our products outside the development labs.                           Integration Deep Dives: Throughout the conference, this year, the focus was on helping customers quickly migrate to the cloud seamlessly. Artificial Intelligence embedded in the technologies that Oracle delivers helps customers automate and innovate quicker, more securely, and with the least disruption to their existing operations. Application integration and Data Integration, two areas that have consistently contributed to the growth of Oracle Customers’ success in the move to the cloud, had their own set of sessions. Here is a list of the different sessions, topics, and labs, that OOW18 hosted around integration. There were customer sessions, product roadmap sessions, thought leadership sessions, and demos, to cater to every information that our current and prospective customers would need to make the best decision to partner with us on their journey to autonomous data warehousing and the cloud. Innovation Awards: No Oracle Open World is complete without the signature red-carpet event, The Oracle Excellence Awards. This year the award winners included major companies and organizations including American Red Cross, Deloitte, Gap, Hertz, National Grid, Sainsbury's and Stitch Fix. While the winners no doubt showcase the best of the use of Oracle technologies, they represent only a small fraction of the best and innovative use of Oracle technologies in the real world. It is always a matter of pride to watch our customers describe, in their own words, the difference they make in turn to their end customers using Oracle technologies.                           Even as the rumbles of this year’s Open World dies down, we at Oracle’s Integration camp are gearing up for exciting new releases and features. Oracle Data Integration is taking on more and more capabilities, baking them into a seamlessly integrated platform. Existing customers will notice how rapidly the changes are coming without feeling the need to learn new skills that bridges various roles with a single data integration platform. New customers will be delighted at the easy-to-use packaging and pricing models. Oracle Application Integration is meanwhile bridging the requirements that arise out of needing applications to be connected. With out-of-the-box connectors, ERP integrations, and features that seamlessly utilize artificial intelligence, Oracle’s Application Integration brings process automation and self-service integration to our customers. Here are just some of the commendations and accolades that Oracle Data Integration and Oracle Application Integration received from the analysts recently. This has been an exceptional Open World, once again reminding me of Oracle’s technologies, deep technical and business expertise, and customer commitment. I am already looking forward to the next OpenWorld. 

The Annual Oracle Tech Bonanza The first time I attended Oracle’s Open World, in 2013, was when I truly understood the scale of innovation and expertise that Oracle brings to its customers. Over the...

GoldenGate Solutions and News

Release Announcement for Oracle GoldenGate 18.1

What’s New in Oracle GoldenGate 18.1 To succeed in today’s competitive environment, you need real-time information. This requires a platform that can unite information from disparate systems across your enterprise without compromising availability and performance. Oracle GoldenGate 18c is a high-performance software application for real-time transactional change data capture, transformation, and delivery, offering bidirectional data replication. The application enables you to ensure that your critical systems are operational 24/7, and the associated data is distributed across the enterprise to optimize decision-making. GoldenGate 18.1 Platform Features For the Oracle Database Oracle Database 18c Support Capture and Delivery support for Oracle Database 18c, cloud and on-premises Autonomous Data Warehouse Cloud (ADWC) and Autonomous Transaction Processing (ATP) Support Easily connect to ADWC and ATP to deliver transactions Identity Column Support Simplified support for handling identity columns in the Oracle Database AutoCDR Improvements Support for tables with unique keys (UK) Composite Sharding Support for multiple shardspaces of data using consistent partitioning In-Database Row Archival Support Support for compressed invisible rows For MySQL MySQL Remote Capture Support Capture MySQL DML transactions from a remote Linux hub.Use for remote capture against MySQL, Amazon RDS for MySQL, and Amazon Aurora MySQL Database running on Linux or Windows. For DB2 z/OS DB 12.1 Support TIMESTAMP w/TIMEZONE and Configurable Schema for Extract’s Stored Procedure For DB2 LUW Cross Endian Support for Remote Capture and PureScale Support For Teradata Teradata 16.20 Support for Delivery Join us in upcoming events: Stay up to date by visiting our Data Integration Blog for up to date news and articles. Save the Date!  Oracle OpenWorld is October 22nd through the 25th. 

What’s New in Oracle GoldenGate 18.1 To succeed in today’s competitive environment, you need real-time information. This requires a platform that can unite information from disparate systems across...

Data Integration

2018 Oracle OpenWorld Data Integration Sessions, Labs and Demos

                                  With OpenWorld 2018 just days away, we can’t wait to welcome you to San Francisco. As you begin thinking of ways your company fits into a data-driven economy, you’ll need to think about how all your business data and cloud data can work together to provide meaningful insights and make better decisions. As our industry continues to roll out new technologies like AI and machine learning, you’ll want to think how your data can work with machine learning systems to get insights from patterns. Learn from the experts how a unified data infrastructure helps you migrate data to a data warehouse, process IoT data with Apache Kafka, and manage data lifecycle for greater transparency. There are over 30 data integration sessions, labs, and demos that showcase Oracle’s data integration technologies. Make room in your schedules to learn from experts who have helped their organizations successfully transform in this digital age. We want to highlight a few sessions here, but there are plenty more that you should plan on attending. Scan the QR-code or click on this link  to explore all the sessions that may interest you.   Oracle’s Data Platform Roadmap: Oracle GoldenGate, Oracle Data Integrator, Governance [PRM4229]   Jeff Pollock, Vice President of Product, Oracle Monday, Oct 22, 11:30 a.m. - 12:15 p.m. | Moscone West - Room 2002 This session explores the range of solutions in Oracle’s data platform. Get an overview and roadmap for each product, including Oracle Data Integrator, Oracle GoldenGate, Oracle Metadata Management, Oracle Enterprise Data Quality, and more. Learn how each solution plays a role in important cloud and big data trends, and discover a vision for data integration now and into the future. Oracle’s Data Platform in the Cloud: The Foundation for Your Data [PRO4230] Denis Gray, Senior Director - Data Integration Cloud, Oracle Monday, Oct 22, 12:30 p.m. - 1:15 p.m. | Marriott Marquis (Golden Gate Level) - Golden Gate C3 The rapid adoption of enterprise cloud–based solutions brings with it a new set of challenges, but the age-old goal of maximizing value from data does not change. Oracle’s data platform ensures that your data solution is built from the ground up on a foundation of best-of-breed data integration, big data integration, data governance, data management, and data automation technologies. As customers ascend more of their enterprise applications to the cloud, they realize a cloud-based enterprise data platform is key to their success. Join this session led by Oracle product management to see how Oracle’s data platform cloud solutions can solve your data needs, and learn the product overview, roadmap, and vision, as well as customer use cases. Oracle Data Integration Platform Cloud: The Foundation for Cloud Integration [THT6793] Denis Gray, Senior Director - Data Integration Cloud, Oracle Tuesday, Oct 23, 5:00 p.m. - 5:20 p.m. | The Exchange @ Moscone South - Theater 5 The rapid adoption of enterprise-cloud-based solutions brings with it a new set of challenges. However, the age-old goal of maximizing value from data does not change. Powered by Oracle GoldenGate, Oracle Data Integrator, and Oracle Data Quality, Oracle Data Integration Platform Cloud ensures that your data solution is built from the ground up on a foundation that is built on best-of-breed data integration, big data integration, data governance, data management, and data automation technologies. Join this mini theater presentation to see the power and simplicity of Oracle Data Integration Platform Cloud. See how it utilizes machine learning and artificial intelligence to simplify data mapping, data transformation, and overall data integration automation.   After all the sessions, JOIN US and unwind at our Oracle Integration & Data Integration Customer Appreciation Event @OOW18, Thursday, October 25, 2018, 6pm-10pm. Barbarossa Lounge, 714 Montgomery Street, San Francisco, CA! Pass Code Needed to Register - #OOW18 Registration link https://www.eventbrite.com/e/oracle-integration-and-data-integration-customer-appreciation-event-oow2018-registration-51070929525   You can start exploring App Integration and Data Integration sessions in the linked pages.  We are also sharing #OOW18 updates on Twitter: App Integration and Data Integration. Make sure to follow us for all the most up-to-date information before, during, and after OpenWorld!

                                  With OpenWorld 2018 just days away, we can’t wait to welcome you to San Francisco. As you begin thinking of ways your company fits into a data-driven economy, you’ll...

Data Integration

Integration Podcast Series: #1 - The Critical Role Integration Plays in Digital Transformation

Authored by Madhu Nair, Principal Product Marketing Director, Oracle Digital transformation is inevitable if organizations are looking to thrive in today’s economy. With technologies churning out new features based on cutting edge research like those based on Artificial Intelligence (AI), Machine Learning (ML) and Natural Language Processing (NLP), business models need to change to adopt and adapt to these new offerings. In the first podcast of our “Integration: Heart of the Digital Economy” podcast series, we discuss, among other questions: What is digital transformation? What is the role of Integration in digital transformation? What roles do Application and Data Integration play in this transformation? Businesses, small and big, are not able to convert every process into a risk reducing act or a value adding opportunity. Integration plays a central role in the digital transformation of a business. Businesses and technologies run on data. Businesses also run applications and processes. Integration helps supercharge these critical components of a business. For example, cloud platforms now offer tremendous value with their Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS) offerings. Adopting and moving to the cloud would help companies take advantage of the best technologies to run their businesses on without having to worry about the costs of building and maintaining these sophisticated solutions. A good data integration solution should allow you to harness the power of data, work with big and small data sets easily and cost effectively, and make data available where data is needed. A good application integration solution would allow businesses to quickly and easily connect application, orchestrate processes, and even monetize applications with the greatest efficiency and lowest risk. Piecemeal cobbling together of so critical elements of digital transformation would undermine the whole larger cause of efficiency that such a strategic initiative aims to achieve. Digital transformation positions businesses to better re-evaluate their existing business models allowing organizations to focus on their core reason for existence. Learn more about Oracle’s Data Integration Solution here. Learn more about Oracle’s Application Integration Solution here. Oracle Cloud Café Podcast Channel Be sure to check out the Oracle Cloud Café, where you can listen to conversations with Oracle Cloud customers, partners, thought leaders and experts to get the latest information about cloud transformation and what the cloud means for your business.

Authored by Madhu Nair, Principal Product Marketing Director, Oracle Digital transformation is inevitable if organizations are looking to thrive in today’s economy. With technologies churning out new...

Webcast: Data Integration Platform Cloud with Autonomous Capabilities - Building an Intelligent Data Integration Platform

Oracle Data Integration Platform Cloud, also referred to as DIPC, brings together years of expertise and vision into a single platform that delivers on the many requirements needed of a data integration solution. DIPC is ambitious in scope and rich in features. DIPC now includes within its platform features underpinned by artificial intelligence, machine learning, and natural language processing. In this webcast, I was joined by Jeff Pollock, Vice President of Product Management for Data Integration Cloud products at Oracle, and Kalyan Villuri, Senior Database Manager at Veritas technologies, LLC. We cover quite a lot of ground, not just about the product, but about best practices for integrating data. Watch this webcast if You are looking for a solution that can bring scalability and trust to your analytics solutions, You are looking to adopt the Oracle Cloud and autonomous data warehousing, You are considering, or are in the middle of any big data projects, You want to see a real-life example of how customers are using Oracle data integration. DIPC unifies a number of Oracle’s flagship technologies under a modern and intuitively designed interface. For replication, change data capture, or real-time data streaming capabilities, DIPC relies on expertise built and expanded in Oracle’s GoldenGate technology as a foundation. Oracle Data Integrator, Oracle’s flagship ETL and bulk data transformation engine, provides robust capabilities that are used as a starting point for DIPC's ETL capabilities. Oracle Enterprise Data Quality and Oracle Stream Analytics engines provide data quality and data streaming capabilities within DIPC. DIPC, however, is not just a repackaging of these mature products. It is a re-imagination of how various best of breed data integration solutions can come together and work seamlessly, finding synergies in their features and elevating smaller piecemeal tasks and projects into a solution based approach. For example, DIPC introduces the concept of “elevated tasks” and “atomic tasks”. The latter, atomic tasks, are equivalent to point tasks that are used to accomplish smaller data requirements and logic, while the former, elevated tasks, consists of end goal oriented (e.g. building a data lake, or prepping data) groupings that bring together often encountered technological requirements into simple and logical task groupings. We are excited to bring DIPC to market at a juncture where Data Integration is gaining more relevance to our customers as they engage in business transformations and other strategic initiatives. To learn more about DIPC watch the webcast here.

Oracle Data Integration Platform Cloud, also referred to as DIPC, brings together years of expertise and vision into a single platform that delivers on the many requirements needed of a data...

Data Integration

#OOW18 Executive Keynotes and Sessions You Won’t Want to Miss

With Oracle OpenWorld 2018 less than two weeks away, you are probably busy crafting an agenda to fit in all the sessions you want to see. We want to make sure your experience is tailored to perfection. In a couple days, we will share our full list of integration sessions and highlight a few special events just for Integration folks. In the meantime, let’s start our planning with a bang by introducing you to some of the executive keynotes and sessions we are most excited about: CLOUD PLATFORM & CLOUD INFRASTRUCTURE EXECUTIVE KEYNOTES AND SESSIONS Cloud Platform Strategy and Roadmap (PKN5769) – Amit Zavery Mon Oct 22, 9-9:45am | Yerba Buena Theater In this session, learn about the strategy and vision for Oracle’s comprehensive and autonomous PaaS solutions. See demonstrations of some of the new and autonomous capabilities built into Oracle Cloud Platform including a trust fabric and data science platform. Hear how Oracle’s application development, integration, systems management, and security solutions leverage artificial intelligence to drive cost savings and operational efficiency for hybrid and multi-cloud ecosystems. Oracle Cloud: Modernize and Innovate on Your Journey to the Cloud (GEN1229) – Steve Daheb Tue Oct 23, 12:30-1:15pm | Moscone West 2002 Companies today have three sometimes conflicting mandates: modernize, innovate, AND reduce costs. The right cloud platform can address all three, but migrating isn’t always as easy as it sounds because everyone’s needs are unique, and cookie-cutter approaches just don’t work. Oracle Cloud Platform makes it possible to develop your own unique path to the cloud however you choose—SaaS, PaaS, or IaaS. Learn how Oracle Autonomous Cloud Platform Services automatically repairs, secures, and drives itself, allowing you to reduce cost and risk while at the same time delivering greater insights and innovation for your organization. In this session learn from colleagues who found success building their own unique paths to the cloud. Autonomous Platform for Big Data and Data Science (PKN3898) – Greg Pavlik Tue Oct 23, 5:45-6:30pm | Yerba Buena Theater Data science is the key to exploiting all your data. In this general session learn Oracle’s strategy for data science: building, training, and deploying models to uncover the hidden value in your data. Topics covered include ingestion, management, and access to big data, the raw material for data science, and integration with autonomous PaaS services. The Next Big Things for Oracle’s Autonomous Cloud Platform (PKN5770) – Amit Zavery Wed Oct 24, 11:15-12pm | The Exchange @ Moscone South - The Arena Attend this session to learn about cutting-edge solutions that Oracle is developing for its autonomous cloud platform. With pervasive machine learning embedded into all Oracle PaaS offerings, see the most exciting capabilities Oracle is developing including speech-based analytics, trust fabric, automated application development (leveraging AR and VR), and digital assistants. Find out how Oracle is innovating to bring you transformational PaaS solutions that will enhance productivity, lower costs, and accelerate innovation across your enterprise.   You can start exploring App Integration and Data Integration sessions in the linked pages. We are also sharing #OOW18 updates on Twitter: App Integration and Data Integration. Make sure to follow us for all the most up-to-date information before, during, and after OpenWorld!

With Oracle OpenWorld 2018 less than two weeks away, you are probably busy crafting an agenda to fit in all the sessions you want to see. We want to make sure your experience is tailored to...

GoldenGate Solutions and News

Oracle GoldenGate Plug-in for Oracle Enterprise Manager v13.2.2.0.0 is now available

We have released GoldenGate OEM Plug-in 13.2.2.0.0. The release's primary focus was to support the monitoring of Oracle GoldenGate 12.3 Microservices (MA) Instances. So now, you can discover and monitor the GoldenGate 12.3 Classic and MA Instances through the GoldenGate OEM Plug-in 13.2.2.0.0. While you discover the GoldenGate MA Instances, you need to specify few details like Service Manager name, password, hostname and port number. We have introduced the new property GoldenGate Classic or Microservices that need to be specified based on what GoldenGate Instances, the Classic or Microservices you would want to discover. You can see the status of the services that you have deployed in your GoldenGate MA Instances. The extract, replicat shows the common metrics available for both classic and MA architecture. In the future release, we will add MA specific metrics and detailed monitoring of MA services. You need to provide the monitoring credentials for Microservices so that you can view the configuration tab/log data and able to start/stop the GoldenGate processes.     The GoldenGate OEM Plug-in has upgraded infrastructure to be compatible with newer version of Enterprise Manager (EM) to 13.2.2.0.0.  You would not require to setup GoldenGate jAgent to communicate with GoldenGate OEM Plug-in for GoldenGate 12.3 MA Instances. The GoldenGate MA architecture provides the RESTful APIs to monitor and manage the GoldenGate MA Instances. The GoldenGate OEM Plug-in uses these RESTful APIs to communicate with GoldenGate MA Instances. For your GoldenGate 12.3 Classic Instances, you would still need to setup GoldenGate jAgent 12.2.1.2.0+ for the communication purposes.   We also have, allowed you to edit Big Data handler properties files from Oracle GoldenGate OEM Plug-In.   You can get more details of the release from the documentation and you may download the software from OSDC, OTN, and EM Store.    We are working to get more features around monitoring the GoldenGate Microservices architecture in future releases. Let me know if you have any questions.

We have released GoldenGate OEM Plug-in 13.2.2.0.0. The release's primary focus was to support the monitoring of Oracle GoldenGate 12.3 Microservices (MA) Instances. So now, you can discover...

Data Integration

New Whitepaper: Oracle GoldenGate - Innovations for Another 20 Years

  Over the past 20 years, the GoldenGate data replication platform has evolved from a startup technology targeted for ATM bank networks to what is now a global phenomenon used in every industry by 1000’s of businesses on every continent of the planet. By most measures, GoldenGate has become the most successful data integration product in the history of enterprise software. What started it all was an intense focus on solving the most demanding business continuity challenges that demand zero-downtime of databases and constant availability of important business data. As the technology advanced further, it became widely used for high-end analytic data warehouses and decision support scenarios for most of the Global 2000 industrial base. After 20 years of being on top, there are a whole new set of innovations that will propel the GoldenGate technology for another two decades of market dominance. These recent innovations include: Non-Relational Data Support – for SaaS Applications, Big Data, and Cloud Kernel Integration with Oracle Database – far better performance than any other vendor Remote Capture for Non-Oracle Databases – reduced workloads and simpler admin Simplification, Automation and Self-Service – no need for DBAs with most actions Microservices Core Foundation – more secure, more modular, and easier to work with Simplified, Open Framework for Monitoring – more choices for DevOps Containers, Kubernetes and Docker – faster and easier to deploy GoldenGate Stream Processing and Stream Analytics – added value with event processing Autonomous Cloud – let Oracle Cloud do the patching and optimizing for you Low-Cost (Pay As You Go) Subscriptions – GoldenGate for the cost of a cup of coffee The remainder of this paper provides more details for these innovations and explains how they will drive business results for the kind of modern digital transformation that IT and business leaders are seeking today. Click here to read the full whitepaper! .  

  Over the past 20 years, the GoldenGate data replication platform has evolved from a startup technology targeted for ATM bank networks to what is now a global phenomenon used in every industry by...

Data Integration

GoldenGate for Big Data 12.3.2.1.1 Release Update

Date: 05-Sep-2018 I am pleased to announce the release of Oracle GoldenGate for Big Data 12.3.2.1.1  Major features in this release include the following: New Target - Google BigQuery Oracle GoldenGate for Big Data release 12.3.2.1.1 can deliver CDC data to Google BigQuery cloud data storage from all the supported GoldenGate data sources. New Target – Oracle Cloud Infrastructure Object Storage Cloud Oracle GoldenGate for Big Data can now directly upload CDC files in different formats to Oracle Object Storage on both Oracle Cloud Infrastructure (OCI) and Oracle Cloud Infrastructure Classic (OCI-C). The integration to Object Storage cloud is provided by File Writer Handler. New Target: Azure Data Lake You can connect to Microsoft Azure Data Lake to process big data jobs with Oracle GoldenGate for Big Data. Other Improvements: Extended S3 Targets: Load files to third Party S3 compatible Object Storages Oracle GoldenGate for Big Data can now officially write to third-party Object Storages which are compatible with S3 API such as Dell-ECS Storage. Support for Kafka REST Proxy API V2 You can now either use Kafka REST Proxy API V1 or V2 and it can be specified in the Big Data Properties file. Security: Support for Cassandra SSL Capture Length Delimited Value Formatter The Length Delimited Value Formatter is a row-based formatter. It formats database operations from the source trail file into a length delimited value output. Timestamp with Timezone Property  You can consolidate the format of timestamp with this timezone property Avro Formatter Improvements You can write the Avro decimal logical type and Oracle NUMBER type. Newer Certifications like Apache HDFS 2.9, 3.0, 3.1 Hortonworks 3.0, CDH 5.15,  Confluent 4.1, 5.0 Kafka 1.1, 2.0 and many more !!! More information on Oracle GoldenGate for Big Data Learn more about Oracle GoldenGate for Big Data 12c Download Oracle GoldenGate for Big Data 12.3.2.1.1 Documentation for Oracle GoldenGate for Big Data 12.3.2.1.1 Certification Matrix for Oracle GoldenGate for Big Data 12.3.2. Prior Releases: May 2018 Release: Oracle GoldenGate for Big Data 12.3.2.1 is released Aug 2017 Release: What Everybody Ought to Know About Oracle GoldenGate Big Data 12.3.1.1 Features

Date: 05-Sep-2018 I am pleased to announce the release of Oracle GoldenGate for Big Data 12.3.2.1.1  Major features in this release include the following: New Target - Google BigQueryOracle GoldenGate...

Oracle Named a Leader in 2018 Gartner Magic Quadrant for Data Integration Tools

Oracle has been named a Leader in Gartner’s 2018 “Magic Quadrant for Data Integration Tools” report based on its ability to execute and completeness of vision. Oracle believes that this recognition is a testament to Oracle’s continued leadership and focus on in its data integration solutions. The Magic Quadrant positions vendors within a particular quadrant based on their ability to execute and completeness of vision. According to Gartner’s research methodologies, “A Magic Quadrant provides a graphical competitive positioning of four types of technology providers, in markets where growth is high and provider differentiation is distinct: Leaders execute well against their current vision and are well positioned for tomorrow. Visionaries understand where the market is going or have a vision for changing market rules, but do not yet execute well. Niche Players focus successfully on a small segment, or are unfocused and do not out-innovate or outperform others. Challengers execute well today or may dominate a large segment, but do not demonstrate an understanding of market direction.” Gartner shares that, “the data integration tools market is composed of tools for rationalizing, reconciling, semantically interpreting and restructuring data between diverse architectural approaches, specifically to support data and analytics leaders in transforming data access and delivery in the enterprise.” The report adds “This integration takes place in the enterprise and beyond the enterprise — across partners and third-party data sources and use cases — to meet the data consumption requirements of all applications and business processes.” Download the full 2018 Gartner “Magic Quadrant for Data Integration Tools” here. Oracle recently announced autonomous capabilities across its entire Oracle Cloud Platform portfolio, including application and data integration. Autonomous capabilities include self-defining integrations that help customers rapidly automate business processes across different SaaS and on-premises applications, as well as self-defining data flows with automated data lake and data prep pipeline creation for ingesting data (streaming and batch). A Few Reasons Why Oracle Data Integration Platform Cloud is Exciting Oracle Data Integration Platform Cloud accelerates business transformation by modernizing technology platforms and helping companies adopt the cloud through a combination of machine learning, an open and unified data platform, prebuilt data and governance solutions and autonomous features. Here are a few key features: Unified data migration, transformation, governance and stream analytics – Oracle Data Integration Platform Cloud merges data replication, data transformation, data governance, and real time streaming analytics into a single unified integration solution to shrink the time to complete end-to-end business data lifecycles.  Autonomous – Oracle Data Integration Platform Cloud is self-driving, self-securing, and self-repairing, providing recommendations and data insights, removing risks through machine learning assisted data governance, and automatic platform upkeep by predicting and correcting for downtimes and data drift. Hybrid Integration –Oracle Data Integration Platform Cloud enables data access across on-premises, Oracle Cloud and 3rd party cloud solutions for businesses to have ubiquitous and real-time data access. Integrated Data Lake and Data Warehouse Solutions – Oracle Data Integration Platform Cloud has solution based “elevated” tasks that automate data lake and data warehouse creation and population to modernize customer analytics and decision-making platforms. Discover DIPC for yourself by taking advantage of this limited time offer to start for free with Oracle Data Integration Platform Cloud. Check here to learn more about Oracle Data Integration Platform Cloud. Gartner Magic Quadrant for Data Integration Tools, Mark A. Beyer, Eric Thoo, Ehtisham Zaidi, 19 July 2018. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.  

Oracle has been named a Leader in Gartner’s 2018 “Magic Quadrant for Data Integration Tools” report based on its ability to execute and completeness of vision. Oracle believes that this recognition is...

Data Integration

New Whitepaper: Leverage the Oracle Data Integration Platform Inside Azure and Amazon Cloud

In this whitepaper, find out how you can leverage Oracle Data Integration Platform Cloud to move your on-premises data onto Azure and Amazon Web Services:   Oracle Data Integration Platform Cloud (DIPC) is a highly innovative data integration cloud service, providing a series of industry-first capabilities have been rolled out since inception in 2017, including streaming data replication, pushdown processing for data transformations (no engine required) ,and first class big data ingestion capabilities that support a wide variety of Apache open source projects such as Hive, HBase, Flume, Cassandra, and Kafka.   One of the most innovative architectural patterns that Oracle Data Integration Platform Cloud supports is the ability to push workloads to compute resources of the customer’s choice while preserving the capability for customers to keep their physical records behind their own firewalls and within their own security zones.   While you can move data to both Oracle and non-Oracle environments, this paper focuses on moving data to Azure and Amazon Clouds. There is absolutely no DIPC requirement for customers to put any of their business data into Oracle networks or any cloud resources at all. Oracle DIPC allows customers to keep their data within any of the following:   On Premise Data Centers – which could include any regional or geographically distributed data centers that customers operate themselves or lease from 3rd party operators Amazon Cloud Data Centers – supporting IaaS and PaaS integrations with Amazon services in any AWS regional data centers Azure Cloud Data Centers – supporting IaaS and PaaS integrations with Microsoft Azure data centers across regions Or any other data center that needs to support their workloads   The remainder of this paper will provide specific details about supported use cases for Oracle DIPC to support innovative next-generation data flows within Amazon and Azure clouds.   Click here to read the full whitepaper.  

In this whitepaper, find out how you can leverage Oracle Data Integration Platform Cloud to move your on-premises data onto Azure and Amazon Web Services:   Oracle Data Integration Platform Cloud (DIPC)...

Data Integration

Data Integration Platform Cloud (DIPC) 18.3.3 New Tasks

Just a few days ago, we wrote about the newest release of Data Integration Platform Cloud (DIPC) 18.3.3.  This is all very exciting!  Now, a few bits in an effort to share a bit more on the two newest Elevated Tasks and inclusion of Stream Analytics to DIPC.   This release of DIPC helps with data lake automation, enabling an intuitive instantiation and copy of data into a data lake, in an effort to help reduce some of the existing data engineer/ data scientist friction through a new Data Lake Builder task.  You can quickly create a comprehensive, end-to-end repeatable data pipeline to your data lake.  And – note that nothing is moved to data lake without being fully governed!  When you add data to the data lake, DIPC follows a repeatable pattern to harvest, profile, ingest, shape, copy, and catalog this data. Data can be ingested from a variety of sources, including relational sources, flat files, etc. Harvested metadata will be stored in the DIPC Catalog, and the data will be transformed and secured within the target data lake for downstream activities.  For more information, see Adding Data to Data Lake.   The Replicate Data Task helps address high availability.  Replicate into Oracle… or Kafka!  And, bring that together with Stream Analytics whereby event process is made possible on real-time data streams, including Spatial, Machine Learning, queries on the data stream or cubes.  With Stream Analytics, you can analyze complex event data streams that DIPC consumes using sophisticated correlation patterns, enrichment, and machine learning to provide insights and real-time business decisions. Very simply, the Replicate Data Task delivers changes from your source data to the target.  You set up connections to your source and target, and from the moment that you run this task, any new transaction in the source data is captured and delivered to the target. This task doesn't perform an initial copy of the source (for the initial load see Setting up a Synchronize Data Task) so you'll get all the changes from the point of time that you started your job. This task is especially ideal for streaming data to Kafka targets.  For more information, see Setting up a Replicate Data Task.   For more tutorials, videos, etc on DIPC – please visit the Documentation, as well as the A-Team Chronicles for interesting Data Integration technical know-how.

Just a few days ago, we wrote about the newest release of Data Integration Platform Cloud (DIPC) 18.3.3.  This is all very exciting!  Now, a few bits in an effort to share a bit more on the two...

Data Integration Platform Cloud (DIPC) 18.3.3 is available!

Data Integration Platform Cloud (DIPC) 18.3.3 is now available! Do you know what DIPC is?  Check out this short 2 minute video!   DIPC is innovative!  With an Elevated Task driven approach that guides users through their Data Integration journey, DIPC seeks to simplify and revolutionize Data Integration!  The Elevated Task is a simple, pre-defined set of steps to assist in creating a specific and useful, common job within Data Integration.  These tasks result in simpler solutions such as data migrations or data warehouse/data lake automation and projects that are delivered more quickly, but yet, well designed and effective. Let’s cover some of the brand new and exciting features and tasks in this release!      This release helps with data lake automation, enabling an intuitive instantiation and copy of data into a data lake, in an effort to help reduce some of the existing data engineer/ data scientist friction through a new Data Lake Builder task!  You can quickly create a comprehensive, end-to-end repeatable data pipeline to your data lake.  And – note that nothing is moved to data lake without being fully governed!      The Replicate Data Task helps address high availability.  Replicate into Oracle… or Kafka!  And, bring that together with Stream Analytics whereby event process is made possible on real-time data streams, including Spatial, Machine Learning, queries on the data stream or cubes.  With Stream Analytics, you can analyze complex event data streams that DIPC consumes using sophisticated correlation patterns, enrichment, and machine learning to provide insights and real-time business decisions. Additionally, you have already heard us mention the Synchronize Data Task, the Data Preparation Task, the ODI Execution Task, but this release features enhancements to many of these! For a quick recap:   Synchronize Data:  The Synchronize Data task enables you to copy your selected data from a source to target, and then keeps both databases in sync. You can also use filter rules to include or exclude specific data entities in your job. Any change in the source schema is captured and replicated in the target and vice versa. After you create a synchronize data task and set it up to synchronize all data or specific data entities, you can run the job. If you setup policies for your job, you'll receive notifications for your specified criteria. For more information, see Creating a Synchronize Data Task.   Data Preparation:  The Data Preparation task enables you to harvest data from a source File or Oracle database Connection, and then cleanse, organize, and consolidate that data, saving it to a target Oracle database. For more information, see Setting up a Data Preparation Task.   ODI Execution:  Invoke an existing ODI Scenario to perform bulk data transformations. For more information, see Setting up an ODI Execution Task.   This release also provides updated components under the covers such as:  Enterprise Data Quality 12.2.1.3.0, Oracle Data Integrator 12.2.1.3.1, GoldenGate for Oracle 11g/12c 12.3.0.1.0, GoldenGate for Big Data 12.3.2.1.0, and GoldenGate for MySQL 12.2.0.1.1.  Oracle Stream Analytics is part of DIPC as well, and is available only for user-managed Data Integration Platform Cloud instances.   Want to learn more?  Visit the DIPC site and check out some of our archived webcasts HERE!  

Data Integration Platform Cloud (DIPC) 18.3.3 is now available! Do you know what DIPC is?  Check out this short 2 minute video!   DIPC is innovative!  With an Elevated Taskdriven approach that guides...

Data Integration

Oracle Integration Day is Coming to a City near You

Are you able to innovate quickly in the new digital world? Are you looking for ways to integrate systems and data faster using a modern cloud integration platform? Is your Data Integration architecture allowing you to meet your uptime, replication and analytics/reporting needs? Is your organization able to achieve differentiation and disruption?  Join Oracle product managers and application/data integration experts to hear about best practices for the design and development of application integrations, APIs, and data pipelines with Oracle Autonomous Integration Cloud and Data Integration Platform Cloud. Hear real-world stories about how Oracle customers are able to adopt new digital business models and accelerate innovation through integration of their cloud, SaaS, on-premises applications and databases, and Big Data systems. Learn about Oracle’s support for emerging trends such as Blockchain, Visual Application Development, Self-Service Integration, and Stream Analytics to deliver competitive advantage. Tampa Integration Day With interactive sessions, deep-dive demos and hands-on labs, the Oracle Integration Day will help you to: Understand how Oracle’s Data Integration Platform Cloud (DIPC) can help derive business value from enterprise data; getting data to the right place at the right time reliably and ensuring high availability Understand Oracle's industry leading use of Machine Learning/AI in its Autonomous Integration Cloud and how it can significantly increase speed and improve delivery of IT projects Quickly create integrations using Oracle’s simple but powerful Integration Platform as a Service (iPaaS) Secure, manage, govern and grow your APIs using Oracle API Platform Cloud Service Understand how to leverage and integrate with Oracle’s new Blockchain Cloud Service for building new value chains and partner networks Integration Day begins on August 8 in Tampa. Register now to reserve your spot! Click the links to learn more about your local Integration Day: August 8, 2018 – Tampa August 15, 2018 – Los Angeles August 22, 2018 – Denver September 6, 2018 – San Francisco September 19, 2018 – New York City September 26, 2018 – Toronto October 3, 2018 – Boston December 5, 2018 – Chicago January 23, 2019 – Atlanta January 30 , 2019 –  Dallas February 6, 2019 – Washington DC February 20, 2019 – Santa Clara  

Are you able to innovate quickly in the new digital world? Are you looking for ways to integrate systems and data faster using a modern cloud integration platform? Is your Data...

New Whitepaper: EU GDPR as a Catalyst for Effective Data Governance and Monetizing Data Assets

The European Union (EU) General Data Protection Regulation (GDPR) was adopted on the 27th of April 2016 and came into force on the 25th of May 2018. Although many of the principles of GDPR have been present in country-specific legislation for some time, there are a number of new requirements which impact any organization operating within the EU. As organizations implement changes to processes, organization and technology as part of their GDPR compliance, they should consider how a broader Data Governance strategy can leverage their regulatory investment to offer opportunities to drive business value. This paper reviews some of the Data Governance challenges associated with GDPR and considers how investment in GDPR Data Governance can be used for broader business benefit. It also reviews the part that Oracle’s data governance technologies can play in helping organizations address GDPR. The following Oracle products are discussed in this paper: Oracle Enterprise Metadata Manager (OEMM)–metadata harvesting and data lineage Oracle Enterprise Data Quality (EDQ)–for operational data policies and data cleansing Oracle Data Integration Platform Cloud–Governance Edition (DIPC-GE)–for data movement, cloud-based data cleansing and subscription-based data governance Read the full whitepaper here.

The European Union (EU) General Data Protection Regulation (GDPR) was adopted on the 27th of April 2016 and came into force on the 25th of May 2018. Although many of the principles of GDPR have been...

Data Integration

Data Integration Platform Cloud for SaaS Applications

Customers generate enormous amounts of data in SaaS applications which are critical to business decisions such as reducing procurement spend or maximizing workforce utilization. With most customers using multiple SaaS applications, many of these decisions are made in analytical engines outside of SaaS, or need external data to be brought to SaaS to make decisions within. In this blog we shall examine common data movement and replication needs in the SaaS ecosystem and how Oracle’s Data Integration Platform Cloud (DIPC) enables access to SaaS data and helps with decision making. Data Integration Challenges for SaaS As applications moved from on-premise to SaaS, while they provided a number of benefits, a number of pre-existing assumptions and architectures changed. Let us examine a few changes in enterprise landscape here, which are by no means comprehensive. First, on-premise applications in most cases provided access to applications at a database level, typically read only. This has changed with hardly any SaaS vendor providing database access. Customers now work with REST APIs (or earlier versions of SOAP APIs) to extract and load bulk data. While APIs have many advantages, including removing dependency on application schema, they are no match for SQL queries and have pre-set data throttling limitations defined by SaaS vendor. Second, most customers have multiple SaaS applications which makes it imperative to merge data from different pillars for any meaningful analysis/insight; Sales with Product, Leads with Contacts; Orders with Inventory and the list goes on. While each of the SaaS applications provide some analytical capability, most customers would prefer modern best of breed tools and open architectures for their data for analytical processing. This could be from traditional relational databases with Business Intelligence to modern Data Lakes with Spark engines.  Third most enterprise customers have either an application or an analytical/reporting platform on-premise, which necessitates data movement between cloud to on-premise; i.e, a hybrid cloud deployment. Fourth, semi-structured and unstructured data sources are increasingly used in decision making. Emails, Twitter feeds, Facebook and Instagram posts, Log files and device data all provide context for transactional data in relational systems.  And finally, decision making timelines are shrinking with need for real-time data analysis more often than not. While most SaaS applications provide batch architectures, and REST APIs they struggle to provide robust streaming capability for real time analysis. Customers need SaaS applications to be part of both Kappa and Lambda style architectures. Let us take a peek into how Oracle Data Integration Platform Cloud addresses these issues.   Mitigating SaaS Data Integration Challenges with DIPC Data Integration Platform Cloud (DIPC) is acloud-based platform for data transformation, integration, replication and governance.  DIPC provides batch and real-time data integration among cloud and on-premises environments and brings together the best of breed Oracle data integration products of Oracle GoldenGate, Oracle Data Integrator and Oracle Enterprise Data Quality within one unified cloud platform. You can find more information on DIPC here. For Oracle’s Fusion applications, such as ERP Cloud, HCM Cloud and Sales Cloud, DIPC supports a number of load and extract methods with out of the box connectors. These include BI Publisher, BI Cloud Connector and other standard SOAP/REST interfaces. The choice of interface depends on specific use case. For example, to extract large datasets for a given subject area (say Financials-> Accounts), BI Cloud Connector (BICC) is ideal with its incremental extract setup in Fusion. BICC provides access to Fusion Cloud data via Public View Objects (PVOs). These PVOs are aggregated into Subject Areas (Financials, HCM, CRM etc), and BICC can be setup to manually or programmatically pull full or incremental extracts. DIPC integrates with BI Cloud Connector to kick off an extract, download the PVO data files in chunks, unzip and decrypt them, extract data from CSV formats, read metadata formats from mdcsv files and finally load them to any target such as Database Cloud Service or Autonomous Data Warehouse Cloud Service. For smaller datasets, DIPC can call existing or custom built BI Publisher reports and load data to any targets. For other SaaS applications, DIPC has drivers for Salesforce, Oracle Service Cloud, Oracle Sales Cloud and Oracle Marketing Cloud. These drivers provide a familiar jdbc style interface for data manipulation while accessing SaaS applications over REST/SOAP APIs. In addition, other SaaS applications that provide JDBC style drivers, such as NetSuite can become a source and target for ELT style processing in DIPC. DIPC has generic REST and SOAP support allowing access to any SaaS REST APIs. You can find list of sources and targets supported by DIPC here. DIPC simplifies data integration tasks using the Elevated Tasks, and users can expect more wizards and recipes for common SaaS data load and extract tasks in future. The DIPC Catalog is populated with metadata and sample data harvested from SaaS applications. In the DIPC Catalog users can create Connections to SaaS applications, and subsequent to which a harvest process will be kicked off and populate the Catalog with SaaS Data Entities. From this Catalog, users will be able to create Tasks with Data Entities as Sources and Targets, and wire together a pipeline data flow including JOINs, FILTERS and standard transformation actions. Elevated tasks can also be built to feed SaaS data to a Data Lake or Data Warehouse such as Oracle Autonomous Data Warehouse Cloud (ADWCS). In addition, there is a full featured Oracle Data Integrator embedded inside for existing ODI customers to build out Extract, Load and Transform scenarios for SaaS data integration.  Customers can also bring their existing ODI scenarios to DIPC using ODITask. ODITask is an ODI scenario exported from ODI and imported into DIPC for execution. ODITask can be wired to SaaS source and targets.   Figure above shows DIPC Catalog populated with ERP Cloud View Objects.   Figure above shows details for Work Order View Object in DIPC Catalog   For Hybrid cloud architectures, DIPC provides a remote agent that includes connectors to a wide number of sources and targets. Customers who wish to move/replicate data from on-prem sources can deploy the agent, and have data pushed to DIPC in the cloud for further processing, or vice versa for data being moved to on-premise applications. The remote agent can also be deployed on non-Oracle cloud for integration with Databases running on 3rdparty clouds. For real-time and streaming use cases from SaaS Applications, DIPC includes Oracle Golden Gate, the gold standard in data replication. When permissible, SaaS Applications can deploy Golden Gate to stream data to external Databases, Data Lakes and Kafka Clusters. Either Golden Gate can be deployed to read directly from the SaaS Production database instance to mine the database redo log files or can run on a standby/backup copy of SaaS database and use the cascading redo log transmission mechanism. This mechanism leads to minimal latency and delivers Change Data Capture of specific SaaS transaction tables to an external database or data warehouse providing real-time transaction data for business decisions. Using these comprehensive features in DIPC, we are seeing customers sync end of day/end of month batches of Salesforce Account information into E-Business Suite. Fusion Applications customers are able to extract from multiple OTBI Subject areas and merge/blend Sales, Financials and Sales / Service objects to create custom datamarts. And in Retail, we have customers using Golden Gate’s change data capture to sync Store data to Retail SaaS Apps at corporate in real time. In summary, DIPC provides a comprehensive set of features for SaaS customers to integrate data into Data Warehouses, Data Lakes, Databases and with other SaaS Applications in both real-time and batch. You can learn more about DIPC here.

Customers generate enormous amounts of data in SaaS applications which are critical to business decisions such as reducing procurement spend or maximizing workforce utilization. With most customers...

Oracle GoldenGate Veridata 12.2+ BP new enhancements

In last week, we have released GoldenGate Veridata bundle patch (12.2.1.2.180615). The release contains the two significant improvements along with few bug fixes. The GoldenGate Veridata certifies the High Availability (HA) for Veridata Server and Veridata Agents. The GoldenGate Veridata is leveraging the High availability support provided by WLS. We officially certified it and had documented the detailed steps for all of you to harness it. You may find the details provided in Oracle By Example created by Anuradha Chepuri.   When primary Veridata server fails down, the other Veridata server(backup or slave server) will serve the requests to connected Veridata agents. All the existing requests need to re-initiate again by users. Both the Veridata servers are connected to the shared repository so that all the metadata are available and updated to both the servers. The Veridata Agent HA support has also been tested, when the primary Veridata agent fails down, the other slave or backup Veridata agent will take over. All the new requests will be diverted to Veridata Agent, and existing requests need to re-initiate by users. The VIP address needs to be added into Veridata Agent configuration file so that seamless fail-over could happen.   The other major feature was to allow Active Directory (AD) users access of GoldenGate Veridata product. The Active Directory users can use the GoldenGate Veridata product. We have created new roles in Veridata for Active Directory.   Following are AD Veridata Roles added:- ExtAdministrator ExtPowerUser ExtDetailReportViewer ExtReportViewer ExtRepairOperator You need to import these roles by creating them in your WebLogic server. In below screen, I have shown how to create the ExtAdministrator role. All other roles can be created similarly. Once, all the required roles are imported in the WebLogic server, and you may assign these roles to your AD users or group of users. Over here, I am editing the ExtAdministration Role. For ExtAdministrator role, I want to add the Group. I am adding the existing AD group called "DIQAdmin" to it.   The AD users who are all part of DIQAdmin can access the Veridata product.   You may see my blog on earlier GoldenGate Veridata release over here. Let me know if you have any questions.

In last week, we have released GoldenGate Veridata bundle patch (12.2.1.2.180615). The release contains the two significant improvements along with few bug fixes. The GoldenGate Veridata certifies the...

Data Integration

Walkthrough: Oracle Autonomous Data Integration Platform Cloud Provisioning

We recently launched Oracle Autonomous Data Integration Platform Cloud (ADIPC) Service, a brand new Autonomous cloud-based platform solution for all your data integration needs that helps migrate and extract value from data by bringing together capabilities of a complete Data Integration, Data Quality, and Data Governance. You can get more information about it in Introducing Oracle Autonomous Data Integration Platform Cloud (ADIPC). In this article, I will focus on the provisioning process and walk you through how to provision Autonomous Data Integration Platform Cloud (ADIPC) Instance in the Oracle Cloud. In the previous blog my colleague Julien Testut has walked you through how to provision the Data Integration Platform Cloud - User Managed, cloud service. Here onward, I will refer "Autonomous Data Integration Platform Cloud" as ADIPC "Autonomous Data Integration Platform Cloud Instance" as ADIPC Instance "Data Integration Platform Cloud" as DIPC User Managed. First, you will need to access your Oracle Cloud Dashboard. You can do so by following the link you received after subscribing to the Oracle Cloud, or you can go to cloud.oracle.com and Sign In from there. The Service does not have any pre-requisite, you can directly create ADIPC Instance. We are providing you the in-built Database Cloud Service (DBCS) Instance for storing the ADIPC Repository content for all Editions. The DBCS Instance will not be accessible to the user, and it is used for internal ADIPC purpose. It is self-managed Instance. Go to the Dashboard, click on Create Instance, click on All Services and scroll down to find Data Integration Platform under Integration. Click on Create next to it. This will get you to the ADIPC Service Console page:   You can create Service by clicking either on QuickStarts or Create Instance. The QuickStarts template will provide you ready to use templates for different Editions. Click on QuickStarts at the right-top corner. The page will display Governance Edition template. The upcoming release will have new templates. The Instance name is automatically generated for you. You may change the name if requires. Click on Create and the ADIPC Instance will be created for you.   If you want to select various input parameters while provisioning, click on Create Instance on Service Console page to navigate to the provisioning screens. In the Details screen, Under Instance Details, enter Service Name, Description, Notification Email, Tags, and On Failure Retain Resources. In Configuration Section, you may select the Region and Availability Domain where you want to deploy your ADIPC Instance. In Service Section, select Data Throughput (Data Volume) that has mainly four choices. The ADIPC has new data volume based metering model, where you choose the option based on your data volume in your Data Integration environment. Your Instance will have the compute resource as per selected data volume.  If you want to utilize your on-premises licenses, you may choose to Bring Your Own License option. In this example, I have selected the Governance Edition that includes Data Governance capabilities in addition to everything included with ADIPC Standard and Enterprise Editions.  When done, click Next to review the configuration summary: Finally, click Confirm to start the ADIPC Instance creation. You can see the status of the new Instance being provisioned in the Oracle Cloud Stack Dashboard. You can also check the Stack Create and Delete History at the bottom of the page. It has more detailed information. You can go to the Dashboard by clicking on Action Menu on left top corner, click Dashboard.   Next, let's customize your dashboard to show ADIPC in the Oracle Cloud Dashboard. From the Dashboard, click on minus - button on right top corner,  then click Customize Dashboard: Scroll down in the list and click Autonomous Data Integration Platform Cloud under Integration section: Autonomous Data Integration will then start appearing on the Dashboard: Click the Autonomous Data Integration Platform Cloud (ADIPC) Action Menu to see more details about your subscription and click Open Service Console to view your ADIPC instances and View Account Usage Details to find out how much data you have already consumed: You can also see the Instance status through ADIPC Service Console. You can click on the instance name to get more details about it. When the provisioning process is over, the ADIPC Instance will show as ‘Ready’: Congratulations! We now have a new Autonomous Data Integration Platform Cloud Instance to work with. You can get more information about it on the product page: Data Integration Platform. In future articles, we will cover more DIPC autonomous capabilities. In the meantime, please write your comments if you have any questions.

We recently launched Oracle Autonomous Data Integration Platform Cloud (ADIPC) Service, a brand new Autonomous cloud-based platform solution for all your data integration needs that helps migrate...

Looking for Cutting-Edge Data Integration & Governance Solutions: 2018 Cloud Platform Excellence Awards

It is nomination time!!!  This year's Oracle Excellence Awards: Oracle Cloud Platform Innovation will honor customers and partners who are on the cutting-edge, creatively and innovatively using various products across Oracle Cloud Platform to deliver unique business value.  Do you think your organization is unique and innovative and is using Oracle Data Integration and Governance?  Are you using Data Integration Platform Cloud, GoldenGate Cloud Service, Oracle Data Integrator Cloud Service, Oracle Stream Analytics, etc?  And are you addressing mission critical challenges?  Is your solution around heterogeneous and global data high availability, database migrations to cloud, data warehouse and data lake automation, low latency streaming and integration, or data governance for the business or IT for example?  Tell us how Oracle Data Integration is impacting your business! We would love to hear from you!  Please submit today in the Data Integration and Governance category. The deadline for the nomination is July 20, 2018.  Win a free pass to Oracle OpenWorld 2018!! Here are a few more details on the nomination criteria: Solution shows innovative and/or visionary use of these products There is a measurable level of impact such as ROI or other business benefit (or projected ROI) Solution should have a specific use case identified Nominations for solutions which are not yet running in production will also be considered Nominations will be accepted from Oracle employees, Oracle Partners, third parties or the nominee company We hope to honor you! Click here to submit your nomination today! And just a reminder:  the deadline to submit a nomination is 5pm Pacific Time on July 20, 2018.

It is nomination time!!!  This year's Oracle Excellence Awards: Oracle Cloud Platform Innovation will honor customers and partners who are on the cutting-edge, creatively and innovatively using...

Data Integration

Oracle GoldenGate for Big Data 12.3.2.1 is released

What’s new in Oracle GoldenGate for Big Data 12.3.2.1 ? New Source - Cassandra Starting Oracle GoldenGate for Big Data release 12.3.2.1, GoldenGate can read from NoSQL data stores. With this release, you will be able to capture changes from Cassandra which is a columnar NoSQL data store. It can also capture from the beginning or also known as Initial Capture.   New Target – Kafka REST Proxy Oracle GoldenGate for Big Data can now natively write Logical Change Records (LCR) data to a Kafka topic in real-time using the REST Proxy interface. Supports DDL changes, Operations such as Insert, Update, Delete and Primary Key Update can be handled. It can support Templates and formatters. It can also provide encoding formats such as AVRO and JSON. It can support HTTPS/SSL layer security   New Target – Oracle NoSQL Oracle GoldenGate for Big Data can now officially write to Oracle NoSQL data stores. It can handle Oracle NoSQL data types, mapping between table and columns, DDL changes to be replicated, Primary key updates. It can support both Basic and Kerberos method of authentication.   New Target – Flat files Oracle GoldenGate for Big Data has a new Flat file writer. This is designed to load to a local file system and then load completed files to another location like HDFS. This means that analytical tools will not try to access the real-time half processed files and also can do post processing capabilities like transform, merge like calling a native function. New Target - Amazon Web Services S3 Storage Oracle GoldenGate for Big Data can create a local file system and then load completed files to another location like AWS S3. S3 handler can write to pre-created AWS S3 buckets or create new buckets using AWS OAUTH authentication method. New Data Formats – ORC & Parquet Oracle GoldenGate for Big Data can write newer data formats such as ORC and Parquet using the new Flat File handler..   Newer Certifications like MapR, Hortonworks 2.7, CDH 5.14, Confluent 4.0, MongoDB 3.6, DataStax Cassandra 5.1, Elasticsearch 6.2, Kafka 1.0 and many more !!! More information on Oracle GoldenGate for Big Data Learn more about Oracle GoldenGate for Big Data 12c Download Oracle GoldenGate for Big Data 12.3.2.1 Documentation for Oracle GoldenGate for Big Data 12.3.2.1 Certification Matrix for Oracle GoldenGate for Big Data 12.3.2.1

What’s new in Oracle GoldenGate for Big Data 12.3.2.1 ? New Source - CassandraStarting Oracle GoldenGate for Big Data release 12.3.2.1, GoldenGate can read from NoSQL data stores. With this release,...

Data Integration

New Releases for Oracle Stream Analytics: Data Sheet Now Available

More than ever before, companies across most industries are challenged with handling large volumes of complex data in real-time. The quantity and speed of both raw infrastructure and business events is exponentially growing in IT environments. Mobile data, in particular, has surged due to the explosion of mobile devices and high-speed connectivity. High velocity data brings high value, so companies are expected to process all their data quickly and flexibly, creating a need for the right tools to get the job done.     In order to address this need, we have released a new version 18.1 of Oracle Stream Analytics (OSA). The product is now available in three capacities: In the cloud as part of Oracle Data Integration Platform Cloud On premise as Oracle Stream Analytics On premise as part of Oracle GoldenGate for Big Data   To try OSA for yourself, you can download it here.   The OSA product allows users to process and analyze large scale real-time information by using sophisticated correlation patterns, enrichment, and machine learning. It offers real-time actionable business insight on streaming data and automates action to drive today’s agile businesses.   Oracle Stream Analytics platform targets a broad variety of industries and functions. A few examples include: Supply Chain and Logistics: OSA provides the ability to track shipments in real-time, alerts for any possible delivery delays, and helps to control inventory based on demand and shipping predictions. Financial Services: OSA performs real-time risk analysis, monitoring and reporting of financial securities trading and calculate foreign exchange prices. Transportation: OSA can create passenger alerts and detect the location of baggage, mitigating some common difficulties associated with weather delays, ground crew operations, airport security issues, and more.   One of the most compelling capabilities of OSA is how it is democratizing the ability to analyze streams with its Interactive Designer user interface. It allows users to explore real-time data through live charts, maps, visualizations, and graphically built streaming pipelines without any hand coding. Data can be viewed and manipulated in a spreadsheet-like tabular view, allowing users to add, remove, rename, or filter columns to obtain the desired result. Perhaps most importantly, users can get immediate feedback on how patterns applied on the live data create actionable results.     A few other notable capabilities include the ability to analyze and correlate geospacial information in streams and graphically define and introspect location data and rules, the predictive analytics availablebased on a wide range of Machine Learning models, and the reusable Business Solution Patterns from which users can select a familiar solution analysis.   Other Data Integration products complement OSA to process information in real-time, including Oracle GoldenGate and Oracle Data Integration Platform Cloud. In particular, Oracle Stream Analytics is integrated with the GoldenGate change data capture platform to process live transaction feeds from transactional sources such as OLTP databases to detect patterns in real-time and prepare and enrich data for analytical stores.   To get a deeper look at the features and functionalities available, check out this new data sheet.   You can learn even more about Oracle Stream Analytics at this product’s Help Center page.  

More than ever before, companies across most industries are challenged with handling large volumes of complex data in real-time. The quantity and speed of both raw infrastructure and business events...

Data Integration

Data Integration Platform Cloud (DIPC) 18.2.3 is Available!

  Data Integration Platform Cloud (DIPC) 18.2.3 is now available!   Here are some of the highlights: DIPC boasts an expanded Intuitive Enhanced User Experience which includes: Data Preparation and Data Synchronization Tasks, and the ODI Execution Task providing better management, execution and monitoring.  There is also a continued focus on overall data integration productivity and ease of use on one cloud platform all within one single pane of glass.   The Synchronize Data Task promises new features to enable schema to schema data synchronization in 3 clicks!  This allows for more control during an initial load and/or during on-going synchronizations.  Better monitoring is also possible.     The new Data Preparation Task provides out of the box end to end data wrangling for better data.  The task allows for simple ingest and harvesting of metadata for easy data wrangling, including integrated data profiling.     The new ODI Execution Task provides the ability to easily execute and monitor Oracle Data Integrator scenarios!  This task supports a hybrid development mode where one can develop or design on-premises to thus import into DIPC and integrate with DIPC Tasks.  This is then all executed and monitored in the DIPC Console.   Additionally, there are also enhancements to the Remote Agent, enabling on-premises to on-premises use cases such as custom Oracle to Oracle replication.  And the Data Catalog provides new data entities and new connections as well, furthering its underpinnings for governance at every enterprise!   Want to learn more?  Visit the DIPC site and check out this short 2 minute video on DIPC!

  Data Integration Platform Cloud (DIPC) 18.2.3 is now available!   Here are some of the highlights: DIPC boasts an expanded Intuitive Enhanced User Experience which includes: Data Preparation and Data...

Oracle Named a Leader in 2018 Gartner Magic Quadrant for Enterprise Integration Platform as a Service for the Second Year in a Row

Oracle announced in a press release today that it has been named a Leader in Gartner’s 2018 “Magic Quadrant for Enterprise Integration Platform as a Service” report for the second consecutive year. Oracle believes that the recognition is testament to the continued momentum and growth of Oracle Cloud Platform in the past year.   As explained by Gartner, the Magic Quadrant positions vendors within a particular quadrant based on their ability to execute and completeness of vision separating into the following four categories: Leaders execute well against their current vision and are well positioned for tomorrow. Visionaries understand where the market is going or have a vision for changing market rules, but do not yet execute well. Niche Players focus successfully on a small segment, or are unfocused and do not out-innovate or outperform others. Challengers execute well today or may dominate a large segment, but do not demonstrate an understanding of market direction.   Gartner views integration platform as a service (iPaaS) as having the “capabilities to enable subscribers (aka "tenants") to implement data, application, API and process integration projects involving any combination of cloud-resident and on-premises endpoints.” The report adds, “This is achieved by developing, deploying, executing, managing and monitoring integration processes/flows that connect multiple endpoints so that they can work together.”   “GE leverages Oracle Integration Cloud to streamline commercial, fulfilment, operations and financial processes of our Digital unit across multiple systems and tools, while providing a seamless experience for our employees and customers,” said Kamil Litman, Vice President of Software Engineering, GE Digital. “Our investment with Oracle has enabled us to significantly reduce time to market for new projects, and we look forward to the autonomous capabilities that Oracle plans to soon introduce.”   Download the full 2018 Gartner “Magic Quadrant for Enterprise Integration Platform as a Service” here.   Oracle recently announced autonomous capabilities across its entire Oracle Cloud Platform portfolio, including application and data integration. Autonomous capabilities include self-defining integrations that help customers rapidly automate business processes across different SaaS and on-premises applications, as well as self-defining data flows with automated data lake and data prep pipeline creation for ingesting data (streaming and batch).   Oracle also recently introduced Oracle Self-Service Integration, enabling business users to improve productivity and streamline daily tasks by connecting cloud applications to automate processes. Thousands of customers use Oracle Cloud Platform, including global enterprises, along with SMBs and ISVs to build, test, and deploy modern applications and leverage the latest emerging technologies such as blockchain, artificial intelligence, machine learning and bots, to deliver enhanced experiences.   A Few Reasons Why Oracle Autonomous Integration Cloud is Exciting    Oracle Autonomous Integration Cloud accelerates the path to digital transformation by eliminating barriers between business applications through a combination of machine learning, embedded best-practice guidance, and prebuilt application integration and process automation.  Here are a few key features: Pre-Integrated with Applications – A large library of pre-integration with Oracle and 3rd Party SaaS and on-premises applications through application adapters eliminates the slow and error prone process of configuring and manually updating Web service and other styles of application integration.  Pre-Built Integration Flows – Instead of recreating the most commonly used integration flows, such as between sales applications (CRM) and configure, price, quoting (CPQ) applications, Oracle provides pre-built integration flows between applications spanning CX, ERP, HCM and more to take the guesswork out of integration.  Unified Process, Integration, and Analytics – Oracle Autonomous Integration Cloud merges the solution components of application integration, business process automation, and the associated analytics into a single seamlessly unified business integration solution to shrink the time to complete end-to-end business process lifecycles.   Autonomous – It is self-driving, self-securing, and self-repairing, providing recommendations and best next actions, removing security risks resulting from manual patching, and sensing application integration connectivity issues for corrective action.   Discover OAIC for yourself by taking advantage of this limited time offer to start for free with Oracle Autonomous Integration Cloud.   Check here for Oracle Autonomous Cloud Integration customer stories.   Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.    

Oracle announced in a press release today that it has been named a Leader in Gartner’s 2018 “Magic Quadrant for Enterprise Integration Platform as a Service” report for the second consecutive year....

Data Integration

How to Increase Productivity with Self-Service Integration

By Kellsey Ruppel, Principal Product Marketing Director, Oracle One of the most exciting innovations in integration over the last decade is arriving just in time to address the surge of productivity apps that need to be integrated into enterprises, including small-to medium-size businesses (SMBs). On a general scale, there are approximately 2,300 Software-as-a-Service (SaaS) apps that SMBs use that need to be integrated. Line of business (LOB) users such as marketing campaign managers and sales managers are looking to perform quick and simple self-service integration of these apps themselves without the need for IT involvement – a huge benefit for SMBs who likely might not have a large IT department to lean on. Oracle Self-Service Integration Cloud Service (SSI) provides the right tools for anyone that wants to connect productivity apps such as Slack or Eventbrite into their SMBs. For example, perhaps you are a Marketing Campaign Manager and want to receive an alert each time a new digital asset is ready for your campaign. Or you are a Customer Support Representative trying to automate the deployment of survey links when an incident is closed. Or maybe you are a Sales Manager who wants to feed your event attendees and survey respondents into your CRM. SSI has the tools to address all these needs and more for your SMB. ​For a comprehensive overview of Oracle Self-Service Integration Cloud Service, take a look at our ebook: Make Your Cloud Work for You. Oracle Self-Service Integration is solving these business challenges by: Connecting productivity with enterprise apps - Addressing the quick growth of social and productivity apps that need to be integrated with enterprise apps. Enabling Self-service Integration - Providing line of business (LOB) users the ability to self-service connect applications with no coding to automate repetitive tasks. Recipe-based Integration - Making it easier to work faster and smarter with modern cloud apps with an easy to use interface, library of cloud application connectors, and ready to use recipes.   To learn more, we invite you to attend the webcast, Introducing Oracle Self-Service Integration, on April 18th at 10:00am PT.   Vikas Anand, Oracle Vice President of Product Management, will discuss: Integration trends such as self-service, blockchain, and artificial intelligence the solutions available in Oracle Self-Service Integration Cloud Service Register today!

By Kellsey Ruppel, Principal Product Marketing Director, Oracle One of the most exciting innovations in integration over the last decade is arriving just in time to address the surge of productivity...

Data Integration

How to Use Oracle Data Integrator Cloud Service (ODI-CS) to Manipulate Data from Oracle Cloud Infrastructure Object Storage

Guest Author:  Ioana Stama - Oracle, Sales Consultant   Introduction This article presents an overview on how to use Oracle Data Integrator Cloud Service (ODI-CS) in order to manipulate data from Oracle Cloud Infrastructure Object Storage. The scenarios here present loading the data in an object stored in Oracle Cloud Infrastructure in a table in Database Cloud Service (DBCS) and then move the object to another storage container. We are going to showcase how ODI-CS can connect to Object Storage Classic with both RESTful Services and CURL commands.   About Object Storage by Oracle Cloud Infrastructure Oracle Object Storage is an internet-scale storage, high performance, and durable storage platform. Developers and IT administrators can use this storage service to store an unlimited amount of data, at a very low cost. With the Oracle Object Storage, you can safely and securely use the web-based console to store or retrieve data directly from the internet or from within the cloud platform, at any time. Oracle Object Storage is agnostic to the data content type. It enables a wide variety of use cases. You can send your backup and archive data offsite, store data for Big Data Analytics to generate business insights, or simply build a scale-out web application. The elasticity of the service enables you to start small and scale your application as needed, and you always pay for only what you use. Oracle Object Storage provides a native REST API, along with OpenStack Swift API compatibility, and an HDFS plug-in. Oracle Object Storage also currently offers a Java SDK, as well as Console and Python CLI access for management. We are going to see how the containers look like in the beginning. The source container: The target container: The target table:   Let’s see first how we prepare the topology for the REST services, the File topology and the database topology. We are going to start with the REST services. We are going to create a connection for the source Object Storage and one for the destination one. Preparing the RESTful Services Topology Go to the Topology tab in ODI Studio and right click on the RESTful Services and select New Data Server. You have to give it a name – eg: TestRest In the REST Service endpoint URL you have to write the endpoint URL of the cloud container. This URL can be built accordingly to Oracle documentation or take it from the cloud dashboard. It is already available there.   In order to connect you have to use the cloud account user as per the below picture. After we save this connection we are going to create two new physical schemas. One for the source Object Storage Container and another one for the destination Object Storage Container. We are creating them under the same data server because both containers are created on the same Oracle Cloud Infrastructure. Right click on the newly created data server and select New Physical Schema. The first one is for the TEXTFILE object in the ODICS container, the source one. The resource URL it is also available in the Cloud Dashboard. Now, we are going to go to the Operations Tab. Here we are going to define some operations in order to manipulate the object. As you can see in the list there are methods from where we can pick a method. We defined operations for deleting, getting and putting the object in the container. Let’s test the service by pressing the Test Restful Service button. A pop-up window will open. Here, by selecting the desired operation, the effective URL is built. We can see here that we can add and modify the parameters. The save request content button opens another pop-up that will give you the chance to select the location where you want to save the content that you get from the object. We are going to do the same for the other container. A new physical schema will be created with another resource URL. In the operations tab we only defined operations for the GET and PUT method. The defined operations for both Objects are defined for the purpose of this demonstration. Preparing the File Topology In the Topology tab in ODI Studio right click on the File topology and select new data server. The host here is the name of the host where we are saving the file. Give it a name eg: JCS File. Please leave the JDBC connection to its default state and the save button. Right click on the File data server created and create a schema. Here we are going to mention the path to the file where the files that we are going to use will be stored. Also we have to mention the path of the folder where ODI is going to create his log and error files. Preparing the Database Topology In the Topology tab in ODI Studio right click on the Oracle Technology and select new data server. Give a name to the connection and specify the database user that you want to connect with. In this case storage is our database user. Please go to the JDBC tab and select the Oracle JDBC driver. Please modify the URL according to your details. The next step is to reverse engineer the database table and the file. After we are going to create a mapping to load data from the file to the table.   Go in the Designer tab and after in the Models tab. Here, click on the arrow and select create new model. We will choose the parameters accordingly to the used technology. For the first model we are going to select Oracle as a technology and DB_Storage_logical as the logical schema. After we do that press save. The next step is to reverse engineer the tables. Click on Selective Reverse-Engineering. Select the FILE_STORAGE table and press the Reverse Engineer button. Now we have to reverse engineer the file. Because the content for this is the same we already have a file created. We are going to create a new model and we will select File as a technology and the logical schema created in the topology tab.   After that, press the save button. Next, right click on the new model and select New Datastore. Give it a name, e.g.: TEXT_DATA and in the resource name tab press on the magnifying glass.  Go to the path where we saved the file (the one mentioned in the physical schema). The next step is to go to the Files tab. Here we have to mention the type of file and the delimitators. File format: Delimited Header (if needed) Record separator: e.g.: Unix Filed Separator: e.g.: Tab.   Press save and go to the Attributes Tab. Here you have to click on the Reverse Engineer Tab.   Preparing the mapping and the packages. Create a new project folder and go to the mappings. Right click and select new mapping. Give it a name: e.g. File_to_Oracle. Here, in the canvas, drag and drop the reversed engineered file and the table. Then connect them with the files as a source and the table as a target. Then press save. The next step is to create the packages. We are going to have two packages. One where we are going to call the RESTful services and one where we are going to call the cURL commands.   RESTful services Right click on the package and select new package. Give it a name. e.g. Flow. Here we are going to use the OdiInvokeRESTfulService component from the tool panel. We are going to use it three times. Once for getting data and save it in a file, then for putting the file in the second Object Storage Container, and the third one to delete the file from the source container. The flow is simple: OdiInvokeRESTfulService for saving the data in the object. The mapping that loads the data in the table. OdiInvokeRESTfulService to put the object in the other container. OdiInvokeRESTfulService to delete the object from the source container. The OdiInvokeRESTfulService has different parameters in the General tab. Here we have to select the operation that we want to use. Also, in the Response File parameter we have to specify the location and the file where we want to save the content of the object. In the command tab we can see the command that is going to be executed when we are going to run the flow. The same applies for the other OdiInvokeRESTfulService commands. Let’s run the workflow.   We can see that the execution was successful. We can see that in the table. Also we can check the containers and see that the object has been moved. The target container: The source container: cURL commands.   We are going to create a new package as we did with the previous one. But from the toolbox we are going to select the OdiOSCommand tool. In the command tab we are going to write the cURL commands that you can find below GET: curl -u cloud.admin:account_password https://identity _domain.storage.oraclecloud.com/v1/Storage-identity_domain/ODICS/TEXTFILE_CURL --output /u01/app/oracle/tools/home/oracle/files_storage/test_data.txt PUT: curl -X PUT -F 'data=@/u01/app/oracle/tools/home/oracle/files_storage/test_data.txt' -u cloud.admin:account_password https:// identity _domain.storage.oraclecloud.com/v1/Storage- identity _domain /ODICS_ARCHIVE/TEXTFILE_CURL DELETE: curl -X DELETE -u cloud.admin:account_password https:// identity _domain.storage.oraclecloud.com/v1/Storage- identity _domain /ODICS/TEXTFILE_CURL The steps are: OdiOSCommand to use cURL to get the content of the object The mapping to load data in the table OdiOSCommand to use cURL to put the new object in the second container. OdiOSCommand to use cURL to delete the object from the source container.   Conclusion Oracle Data Integrator Cloud Service (ODI-CS) is able to manipulate objects in Oracle Cloud Infrastructure Classic. You can leverage on ODI-CS capabilities on using RESTful Services and also commands written in any language in order to integrate all your data.

Guest Author:  Ioana Stama - Oracle, Sales Consultant   Introduction This article presents an overview on how to use Oracle Data Integrator Cloud Service (ODI-CS) in order to manipulate data from Oracle...

Data Integration

Oracle GoldenGate Adapters for Base24 version 12c is released

Oracle GoldenGate Product Management is pleased to announce the release of Oracle GoldenGate Adapters for Base24 version 12c. Oracle GoldenGate Adapters for Base24 version 12c is available for both HPE Integrity NonStop Itanium and HPE Integrity NonStop x86 chipset architectures. GoldenGate with BASE24 offers comprehensive data movement and management solutions by enabling real-time data capture and delivery between processing systems. Oracle GoldenGate for Base24 Adapters 12c contains three modules for Active-Active data synchronization operations. They are as follows: D24 : This enables bi-directional, real-time transactional data synchronization of customer and transaction data continuously, also called as Active-Active configuration. In the event of an outage on one system, D24 processes the full transaction load on the remaining machine, ensuring continuous availability and no data loss. N24: This coordinates the notifications associated with full refresh processing and eliminates operational labor and co-ordination of loading newly refreshed files into production between two BASE24 sites. T24: This helps to move structured (tokenized/segmented) data from BASE24 to heterogeneous targets and formats (file formats or databases or big data). Additional information:  Download Oracle GoldenGate Adapters for Base24 from My Oracle Support. Note: Look for Patch # 27024312 for HP NonStop (Guardian) Itanium and HP NonStop (Guardian) on x86. Documentation for Oracle GoldenGate for Adapters Base24. Certification Matrix for Oracle GoldenGate for Adapters Base24 12c for any platform support clarification. For more information on Base24 or ACI products, you may contact ACI Support.

Oracle GoldenGate Product Management is pleased to announce the release of Oracle GoldenGate Adapters for Base24 version 12c. Oracle GoldenGate Adapters for Base24 version 12c is available for both...

Data Integration

Synchronize Data between Source and Target in 2 Clicks!

The recently launched Data Integration Platform Cloud (DIPC) provides capabilities for various data integration requirements covering data transformation, integration, replication and governance. DIPC introduces the concept of Elevated Tasks to hide the complexity of underlying data integration processes used for achieving an end-to-end use case. Synchronize Data is the first such elevated task that allows you to synchronize a source schema with a target schema with no effort. Synchronize Data allows you to keep data in your target database schema in sync with a production database so that the target database can be used for real time Business Intelligence and reporting without affecting production system performance. Let us quickly understand the challenges in implementing such a data synchronization solution. The entire process can be achieved in two steps. First perform initial load of the existing source data to target and then configure a replication process to replicate all the ongoing transactions to the target. Let’s understand  the steps required if you were implementing it using Oracle Data Integrator (ODI) and Oracle GoldenGate (OGG). Create ODI mappings for each of the tables in the source schema Create and run an ODI procedure to retrieve the System Change Number (SCN) from the database. The data up to this SCN will be loaded by initial load in ODI and all transactions after this SCN will be replicated by OGG Run OGG extract process to capture transactions from the source database Run the ODI mappings created in step 1 to perform initial load up to the SCN Run OGG pump to push trail files Run the replicat process to start applying transactions from SCN checkpoint You notice that there are a number of intricate steps involved here that must be performed in the right order, across different products and requires a handshake between ODI and OGG. To implement all of this, you would require deep understanding of both ODI and OGG and it may take several days if not weeks to achieve the process end to end. Additionally, monitoring the progress of each of the steps and getting consolidated statistics will be another challenge. With the new Synchronized Data task in DIPC, this entire operation is now done with few clicks without worrying about the complexity of the underlying steps. All you need to do is create a Synchronize Data task, which entails having to specify the source and target schema, and run it.  DIPC takes care of creating the appropriate ODI scenarios, retrieving SCN in ODI, passing SCN  from ODI to OGG, and initializing and running relevant OGG processes – OGG extract, OGG pump, OGG replicat. DIPC also provides central monitoring capability so that you can view the ongoing progress of each of the steps and their statistics. Let us go through the steps for synchronizing data to see how easily you can do it in DIPC. First go to the DIPC home page and click on the create Synchronize Data Task On the Task creation screen enter the source and target information and click “Save and Run”. DIPC will save the task and kick off the execution. As part of execution DIPC will perform following operations Create ODI scenario to create tables in target schema and perform initial load Retrieve the System Changes Number (SCN) from the database Run OGG extract process to capture transactions from the source database Run the ODI scenario to perform initial load up to the SCN Run OGG pump to push trail files Pass SCN retrieved by ODI to GG Replicat process Start Replicat process to apply transactions from SCN Congratulations! You have created and executed the Synchronize Data Task. You can see the corresponding Job status on the Jobs page Click on the Job to see status and statistics of different steps. It provides you individual process level and consolidated statistics on inserts, updates, duration and lag. You can also view details on the underlying process that is executed for each step As shown above, DIPC has drastically simplified the data synchronization use case between two databases. Now, anybody can implement such an end-to-end scenario without needing the deep expertise previously required or the  juggle of multiple underlying products. Stay tuned for upcoming blogs on other exciting features introduced in DIPC. Meanwhile, check the product blogs to get more information: Data Integration Platform Cloud and Getting a Data Integration Platform Cloud (DIPC) Trial Instance.

The recently launched Data Integration Platform Cloud (DIPC) provides capabilities for various data integration requirements covering data transformation, integration, replication and governance. DIPC...

Data Integration

Connecting the Dots Between Data and Artificial Intelligence

Podcast: Connecting the Dots between Data and AI Artificial Intelligence Among the many definitions of Artificial Intelligence (AI) there is one trait that is never compromised. That the “intelligence” should always grow and never be static. In other words, the decision-making ability of any AI platform should keep learning and become more sophisticated. As the AI platform encounters more data, it keeps refining its decision algorithm. This process, usually called training the AI, is one of the trickier and exciting part of putting together an AI solution. The more data the AI model encounters, the more the AI platform is trained; which in turn makes the decisions more relevant and real world like. The best solutions that incorporate AI absorb all the data they are exposed to, sifting through them to pick and choose those that are relevant and can add value to maximizing the probability of fulfilling their reason to exist, be it surfacing the next best TV show to watch or sending an emergency signal to a maintenance control room for preventive case of the machine part. Data From All Over to Train The AI Data is crucial to train the AI models. Data Integration provides the necessary technologies to access the data that is required to successfully maintain and grow Artificial Intelligence solutions. Data Access Even at first glance, the volume and variety of data is mind boggling. Just the initial challenge of how to get a handle on the different types and sources of data becomes a challenge of scale and complexity. There is data that is being produced by machines (log data), there is data being produced by humans and healthcare devices. Then there is data that is being generated by business systems. There are video files, audio formats, JSON files, structured and unstructured and semi structured data. The list goes on and on. Data Integration helps with accessing data from all sources onto the platform of choice where the AI “brain” sits, refining and making decisions. Data Latency and Deep Learning Without getting too technical, there are two other important considerations to make AI more powerful. The first consideration is how recent and up-to-date is the information the AI uses to make decisions. Real-time data streaming capabilities fulfil this need. The second consideration is the ability to mine, transform and iteratively move and sift through large data sets. This comes from classic data migration and transformation capabilities. Both these capabilities together combine to stream, feed, and extract insights from data for the AI models. For more information on how to ensure your AI powered platforms and devices have the best access to data, read more about Data Integration Platform Cloud here.

Podcast: Connecting the Dots between Data and AI Artificial Intelligence Among the many definitions of Artificial Intelligence (AI) there is one trait that is never compromised. That the...

Data Integration

Data Integration Platform Cloud (DIPC) Home Page Navigation

Guest Author:  Jayant Mahto - Senior Manager, Data Integration Product Management - Oracle   Home Page Navigation The recent release of Data Integration Platform Cloud (DIPC) provides an easy access to data integration tasks by hiding the complexity with an easy to use interface. The home page layout has been created with this ease of use in mind. All the functions for data transformation, integration, replication and governance are accessible with easy navigation within a few clicks.   Home Page Layout Let us look at the different regions in home page: Top blue bar is for quickly accessing Data Integration actions. Notifications and user information is available at top right area. Left bar is used for quickly jumping to different areas for further details. Bottom Left panel shows high level information quick health check. Bottom Right panel gives further details for the item selected in Left panel.   Now Let us look at these areas for more details. Quick Jump to Home Page Clicking on the Home icon on top left takes user to the home page layout at any time. The top blue bar is used for quickly accessing the Data Integration features. Summary and Details Panel The Summary panel gives you high level information for Agents, Connections and Tasks. This is very useful to the user to get a quick health check for the Data Integration environment. By clicking on the summary information, users can get further details on the right side. In the following example the connection list is shown on the right side. By clicking on the details, user can jump to the corresponding page. In this example user clicks on one of the connections and jumps to the connections page. Note the left arrow next to the connection name. Clicking on it will take the user back to the previous page. Similar navigation to go back is available in other pages as well. Notifications It provides alerts showing number of notifications. Clicking on the notification takes user to the notification page. Clicking on the notifications will take the user to the jobs detailed page for further investigation. Notifications are based on the Job Policies rules. Refer to the section below for Policy creation. Policy Creation for Notification In order to receive notifications, user needs to create policies with notification rules. Policies can be created from the top blue bar or left navigation bar in the Home page.  The following information is needed for the policy: Name: Name of the policy appears in the notification indicating which policy is responsible for the notification Description:  Brief description of policy Severity:  High/Medium/Low. It is used in sorting/filtering the notification Enable: Policy is in effect only if it is enabled. Sometimes policy is created and not enabled unless needed. Policy Metrics: Multiple conditions with option to satisfy all or any condition before notifications are generated.  In the following example, notifications will be generated if Job fails or runs for more than 30 minutes.     Stay tuned for upcoming blogs on other exciting features introduced in DIPC. Meanwhile, check the product blogs to get more information: Data Integration Platform Cloud and Getting a Data Integration Platform Cloud (DIPC) Trial Instance.

Guest Author:  Jayant Mahto - Senior Manager, Data Integration Product Management - Oracle   Home Page Navigation The recent release of Data Integration Platform Cloud (DIPC) provides an easy access to...

Data Integration

Using Oracle Data Integrator (ODI) for Big Data to Load CSV Files Directly into HIVE Parquet Format

Guest Author:  Ionut Bruma - Oracle, Senior Sales Consultant   Introduction This article presents an overview of how to use Oracle Data Integrator (ODI) for Big Data with Hive parquet storage. The scenario shows how we can ingest CSV files into Hive and store them directly in Parquet format using standard connectors and Knowledge Modules (KMs) offered by Oracle Data Integrator for Big Data. For those new to this extension of ODI, Oracle Data Integrator for Big Data brings advanced data integration capabilities to customers who are looking to implement a seamless and responsive Big Data Management platform. For the practical scenario we have used Big Data Lite VM. This machine can be downloaded from OTN following the link attached: http://www.oracle.com/technetwork/database/bigdata-appliance/oracle-bigdatalite-2104726.html   Overview of Hive The Apache Hive software projects a structure over the large datasets residing in distributed storage thus facilitating querying and managing this data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL. Let me walk you through how to set up ODI for this use case. Preparing the File Topology Create a File Data Server: Use the Topology tab of ODI Studio. Right click File technology and click on New to create a new Data Server In the Definition panel, enter a Name: eg: CSV Source In the JDBC panel, leave the default driver details:             JDBC driver: com.sunopsis.jdbc.driver.file.FileDriver             JDBC driver URL: jdbc:snps:dbfile Press  button. A pop-up should show that the connection has been successful as below. Go back to the Topology and right click the newly created Data Server. Choose New Physical Schema. The new configuration window will appear. In the Definition tab enter the connection details. Directory (Schema): <fill_path_to_the_folder_containing_csv_files>. E.g: /home/oracle/movie/moviework/odi/Flat_Files Directory (Schema): repeat the same string here. Press Save. An information screen will pop-up. Press ok. Then expand the Logical Architecture tab and find the File technology. Right click on it and click New Logical Schema to associate it with the newly created Physical Schema. In the Definition tab select the Context (Global in our case, but it can be Production, Development and so on). Associate the context with the Physical Schema previously created. Press Save.     Creating the Model reflecting CSV structure Go to Designer window and expand Models, right click on the  folder and click New Model. In the Definition tab fill in the Name, Technology and Logical Schema. For the Technology choose File, and then choose the Logical Schema you have just created for this technology. Click  button to retrieve all the files. You should see them under the model created. In order to have a preview of the data inside the CSV file, right-click on the datastore and choose View Data: Similar to the creation of the File Data Server in the Topology tab, create a Hive Data Server. Fill in the Name as Hive for example. Under the Connection specify the user (the authentication will be internal to the Hive engine, we will specify password=default in the driver specifications). Fill in the metastore URI and go to the JDBC tab. In the JDBC tab fill in the correct driver and URL for the Hive technology as shown in the following example. We have everything emulated inside Big Data Lite environment [JT1] so we will use localhost and the password default (for default authentication).   Click . The connection test should be successful. Now out of the new Hive Data Server (that’s actually the connection to the server) we will create a new schema or in ODI language a new Physical Schema. Right click on the data server and choose new Physical Schema. Fill in the details related to the schema (the schema should exist in ODI already) that you want to include in the Topology. You can use the same schema for both Work and Staging schema.   Next, expand the Logical Architecture window and locate the Hive technology. Right click on it and create a new Logical Schema that will be linked to the Hive schema. Give it a name and associate the Logical Schema with the Physical Schema as shown below (a dropdown menu is available under Physical Schemas area):   Creating the Model reflecting Hive structure If using reverse engineering then a table should exist in Hive. Here is a table creation DDL example: CREATE TABLE Test (ID int, Name String, Price String) STORED as PARQUET;   Log into Hive and run this code. From the OS command line, run bee command like shown below. Set the database where you want to deploy: Return to ODI Studio, go to Designer window and expand Models, right click on the  folder and click New Model. In the Definition tab fill in the Name, Technology and Logical Schema. For the Technology choose Hive, then choose the logical schema you have just created for this technology. Next, go to Reverse Engineer tab and choose the radio button for Customized. Choose the proper knowledge module as RKM Hive and click . Optionally you can choose a mask for the tables you want to reverse engineer or you will reverse the entire schema. The reverse engineering process can be monitored inside Operator Navigator: If you need to forward engineer your Hive Datastore, you will need to create it and fill it out manually. Right click the Hive data model and click New Datastore. Fill in the Name with any name. For the Datastore Type pick Table. Navigate to Storage attributes and fill in the Table Type: Managed, Storage Type: Native, Row Format: Built-In, Storage Format: PARQUET as shown in the picture below. Navigate to Attributes tab and fill in the attributes of the table:  Next, create the mapping that will convert automatically the CSV file into a Hive Parquet stored structure. Click the project you need to create the mapping in, expand the Folder and right-click on the mappings. Click on New Mapping. Give it a name, and start building the mapping. Our mapping is a one to one mapping because we only load the data into Hive but you can also perform any transformation ODI is capable of before loading the data into Hive Parquet table.   Navigate to Physical tab  and make the following adjustments. Select the access point (MCC_AP in my case) and configure the Loading Knowledge Module as shown in the following picture: It is important that LKM File to Hive LOAD DATA is used (and NOT LKM File to Hive LOAD DATA Direct). In addition, the FILE_IS_LOCAL (outside the cluster) option must be set to True, otherwise it will look for the file in HDFS instead of taking the CSV file from outside the cluster. Configure the Integration part (IKM settings): Select the target table (this should be the Hive Parquet one): Open the Integration Knowledge Module tab and choose the IKM Hive Append.GLOBAL KM. Next, if the table does not exist (you use the forward engineering approach described above) then set the CREATE_TARG_TABLE option to TRUE otherwise keep it false. If you want to truncate the table after each run then set the TRUNCATE option to TRUE. Run the mapping and review the results in Operator Navigator:   Examine the Hive Table: Look inside the Parquet file to see the data: Data_lake.db is the name of my database and mccmnc is the name of my table. You should replace them with your own names.   Conclusion ODI offers out of the box integration with Big Data technologies such as Hive, Hadoop, Spark, Pig, etc. When building a data lake or a data warehouse many files come as flat files in different formats like CSV, TXT, JSON and have to be injected in HDFS/HIVE in formats like Parquet.  ODI is able to build a reusable flow in order to automatically transfer the CSV files as they come from sources directly into the target HIVE tables.  This post presents a step-by-step guide on how to setup the right topology that reflects bots CSV source and HIVE Parquet table target. Then we tried to show two ways of building the target model (reverse engineering an existing table or forward engineer a structure created in ODI) and furthermore how to choose the proper Knowledge Modules for the loading and integration part.                                                                           

Guest Author:  Ionut Bruma - Oracle, Senior Sales Consultant   Introduction This article presents an overview of how to use Oracle Data Integrator (ODI) for Big Data with Hive parquet storage. The...

Data Integration

How to get IDCS OAuth details?

Some of you might not be aware of the process to get the OAuth keys for configuring Data Integration Platform Cloud (DIPC) with On-Premises agents. Keeping this in mind, I have given stepwise details including screenshots to get the OAuth keys for configuring Data Integration Platform Cloud (DIPC) with On-Premises agents. You would need the following four parameters for connecting an on-prem agent to DIPC server with OAuth authentication: idcsServerUrl agentIdcsScope agentClientId agentClientSecret ​You will automatically get the idcsServerUrl and idcsscope when a DIPC instance is provisioned. However you will not have the ClientID and ClientSecret parameter to configure the agent. For getting the ClientID and Client Secret, you would need to do the following three steps as mentioned below:  1. Login to Oracle Identity Cloud Service (IDCS) Console a) Use your browser to go to 'http://cloud.oracle.com' b) Click on 'Sign-in' c) Select 'Cloud Account with Identify Cloud Service' and click "My Services" d) Click "Users" Icon in the top right e) Click on  "Identity Console" to access the Identity Console idcsServerUrl Parameter: Note the URL address for the identity console (up to the oraclecloud.com) For example – https://idcs-xXXxxXXxxNNxxXXXxxxXX.identity.oraclecloud.com f) Login to IdCS 2. Create Trusted Application a. Click on 'Applications' Tab within the IDCS Menu b. Click ' + Add' and Select 'Trusted Application' c. Provide a Name for the Trusted Application (in newer screens it is called "Confidential Application") . Click 'Next'.  Note: All other parameters need not be necessarily filled. d. Select 'Configure this application as a client now' and then check Grant Permissions for 'Resource Owner' , 'Client Credentials', 'JWT Assertion', and  'Refresh Token'. e.  Click to 'Add Scope' in the below section. Select the DIPC Application URL that you have been provisioned. Note: This URL will be used as the idcsScope parameter.   f. Trusted 'Application' would be listed in the scope g. Click 'Next' and then select 'skip for later'  and click 'Next' again in the 'resources' sub-section h. Click 'Finish' in 'Authorization' sub-section i. This will list the trusted application with 'Client ID' and 'Client Secret'. Note down the values down for ClientID and Client Secret for future reference. This will be used later for authentication of your DIPC Remote authentication.   3. Activate the newly created 'Trusted Application' a. Select the Trusted Application from the Application Menu b. Click on 'Activate' button c. Click on 'Activate Application' Confirmation button in the pop-up box d. You will get a confirmation message once the application is activated. Note: In case you need to regenerate the ClientID and ClientToken, you would need the open the Trusted Application and click on button 'Generate Access Token' e. The new application will be listed as a 'Trusted Application' in the 'Applications' Section This step completes the creation of an IDCS OAUTH Application creation and the credentials can be used in other Oracle Cloud Applications.   Reference and Additional Information DIPC Product Documentation Homepage Setting up DIPC Agents to understand what each configuration parameter means or to understand how to manually edit the configuration file to change port number or heartbeat interval Hybrid Data Integration in 3 Steps using Oracle DIPC Blog DIPC certification matrix to know the supported versions of Operating Systems and Source/Target Applications.    

Some of you might not be aware of the process to get the OAuth keys for configuring Data Integration Platform Cloud (DIPC) with On-Premises agents. Keeping this in mind, I have given stepwise details...

Data Integration

Hybrid Data Integration in Three Simple Steps using Oracle Data Integration Platform Cloud (DIPC)

Background We are currently living in a real-time world and data movement is expected to happen real-time across the various enterprise systems. Enterprises are having serious challenges in data integration between on-premises systems and cloud systems. Integration is never an easy problem to solve considering the complexity in handling the knowledge in various complex applications and also managing various data risks for the organization across these hybrid systems. Oracle Data Integration Platform Cloud  (DIPC) has simplified the complexity of data integration with a simplified concept of using an on-premises agent to integrate to any type of data sources or targets. An on-premises DIPC Agent can be broadly used in the following two data integration scenarios: a. On-Premises to Cloud Data Integration and Vice versa ​ In this integration, a DIPC agent is installed in the On-premises data center which communicates to DIPC Host on the Oracle cloud. In this scenario, the data gets transferred from DIPC Agent to the DIPC Host and then to the Oracle Database Cloud Service.  b. On-Premises to On-Premises Data Integration In this integration, a DIPC agent is installed in the On-premises data center which communicates to DIPC Host on the Oracle cloud only for manageability aspects. In this scenario, the data gets transferred from Source Oracle database to Target Oracle database through the DIPC Agent. No customer data will be transferred out of on-premises to Oracle Cloud. 3 Simple Steps for Hybrid Data Integration  In this section, you will learn how to a) Download and Install, b) Configure and c) Run a DIPC agent on your on-premises system to integrate across On-Premises or Hybrid systems. Step 1: Download and Install DIPC Agent i. Log on to Data Integration Platform Cloud. ii. From the left navigation pane, click Agents. iii. From the Download Installer menu, select the available agent package based on the operating system. iv. In the Download Agent Installer window, click OK. v. Once the agent is downloaded, you would need to unzip into directory where you are expecting the agent to run. Note: This location will be called as  ${AGENT_UNZIP_LOC} in the further section vi. This completes your DIPC Agent download and installation step. Step 2 : Configuring On-Premises Agent i. You can find the agent configuration script in the following directory Command> cd [Agent_unzip_dir]/dicloud You will  find scripts dicloudConfigureAgent.bat (on Windows ) and dicloudConfigureAgent.sh (on Unix) ii. You may configure the agent in two ways : a) Basic AuthType  : For Configuring in Basic AuthType, run the following  Command> ./dicloudConfigureAgent.sh [agent_name] -dipchost=[dipc.example.host.com] -dipcport=[port_no] -user=[diuser] -password=[dipassword] -authType=BASIC eg: dicloudConfigureAgent.bat dipcagent001  -recreate -dipchost=135.169.165.107 -dipcport=9073 -user=weblogic -password=password123 -authType=BASIC b) OAuth Type: For Configuring in OAuth Type, run the following Command> ./dicloudConfigureAgent.sh [agentInstanceDirectory] -recreate -debug -dipchost=[dipc.example.host.com] -dipcport=[port] -user=[diuser] -password=[dipassword] -authType=[BASIC/OAUTH2] -idcsServerUrl=[idcs_server url] -agentIdcsScope=[agent_IDCS_Client_Scope] -agentClientId=[Agent_IDCS_clientID] -agentClientSecret=[Agent_IDCS_clientSecret] Eg: ./dicloudConfigureAgent.sh -dipchost=dipcsfungerc4-dipcsv2meteri01.uscom-central-1.c9dev1.oc9qadev.com -dipcport=443 -idcsServerUrl=https://idcs-b8bdf957678a4d91b80801964c406828.identity.c9dev1.oc9qadev.com -agentIdcsScope=https://235C1F73C5B54068A7C02A202D6B2B42.uscom-central-1.c9dev1.oc9qadev.com:443external  -user=firstname.lastname@ORACLE.COM -password=Password123# -agentClientId=fb035f53b22a450982ec22551cccfdd6 -agentClientSecret=62eda7dc-1991-47ca-8205-fde062c62ab8  Note a: You would need to get the OAuth tokens from Oracle Identity Cloud Service (IDCS) prior to configuring DIPC Agent in OAuth mode. Refer to blog  https://blogs.oracle.com/dataintegration/how-to-get-idcs-oauth-details for step-by-step details. Note b: If you need to configure Remote agent with Autonomous DIPC, then you would need to create a Trusted Application using the Oracle IDCS Admin Console and assign "OCMS App" in allowed Scope. iii. This completes DIPC agent configuration  Step 3 : Starting the Agent i. Go to the DIPC agent instance directory Command> cd [AGENT_UNZIP_LOC]/dicloud/agent/[agent_name]/bin/ ii. To Start DIPC Agent:  Command> ./startAgentInstance.sh  iii. If you need to Stop DIPC Agent for any maintenance, you can run the following Command> ./stopAgentInstance.sh Note: If you are using Autonomous DIPC, the you would need to import the certificate from browser into cacerts present in DIPC Agent's JAVA_HOME directory and then run keytool import command. For eg: $JAVA_HOME/bin/keytool -import -alias ocptest -keystore  /u01/JDK/jdk1.8.0_171/jre/lib/security/cacerts -file /scratch/ocp_test.crt Reference and Additional Information DIPC certification matrix to know the supported versions of Operating Systems and Source/Target Applications. DIPC Documentation Homepage Setting up DIPC Agents to understand what each configuration parameter means or to understand how to manually edit the configuration file to change port number or heartbeat interval

Background We are currently living in a real-time world and data movement is expected to happen real-time across the various enterprise systems. Enterprises are having serious challenges in...

GoldenGate Solutions and News

Oracle GoldenGate Studio 12.2.1.3+ new enhancements

In series of blogs around Oracle GoldenGate Foundation Suite (OGFS) products, last time, I mentioned the features launched for GoldenGate Veridata. I am happy to announce that we have released the latest Oracle GoldenGate Studio Bundle patch(12.2.1.3.180125) in last week. The GoldenGate Studio Bundle patch contains the certification of Oracle GoldenGate 12.3 Classic architecture and new CDC capture for SQL Server. Now you can design and deploy your GoldenGate artifacts to GoldenGate 12.3 Classic architecture. The GoldenGate Studio will discover the OGG 12.3 Classic Instance automatically, and you can deploy your existing designs to OGG 12.3 Classic Instances. GoldenGate Studio can help you upgrade from older OGG versions like 12.1, 12.2 to OGG 12.3 Classic. GoldenGate Studio can also help you build OGG 12.3 Classic Instances using Reverse engineering feature. It allows you to simulate your existing replication environment into GoldenGate Studio. For more information about Reverse engineering feature, please read my blog. Oracle GoldenGate for SQL Server team has recently released the new CDC capture for SQL Server. The Oracle GoldenGate Studio supports this new CDC capture for SQL Server, and it will allow you to design, deploy GoldenGate SQL Server including new CDC capture artifacts in Studio. Lastly, this bundle patch contains support altering extract and replicat processes using GoldenGate ALTER commands. Let me know if you have any questions. 

In series of blogs around Oracle GoldenGate Foundation Suite (OGFS) products, last time, I mentioned the features launched for GoldenGate Veridata. I am happy to announce that we have released the...

Data Integration

How to Succeed with Data Ingestion: 5 Best Practices

Data is the starting point of every smart business decision. Virtually every improvement a business makes these days – anything from sales strategies to IT processes to web content and so on – is backed by data that tells a story. With the emergence of big data, businesses have access to huge volumes of structured and unstructured data. The challenge is learning how to use this data to its full potential. In order for data to be used to its full potential, data must always be available in any format, at any time, in any place you need it and it must be governed to ensure trust and quality. Data must be ingested properly then go through an ETL (Extract, Transform, and Load) Pipeline in order to be trustworthy and, if any mistakes are made in the initial stages, they pose a major threat to the quality and the integrity of the output at the end of the process. Data ingestion is the first step in the Data Pipeline. It is the process of moving data from its original location into a place where it can be safely stored, analyzed, and managed – one example is through Hadoop. As companies adjust to big data and the Internet of Thing (IoT), they must learn to grapple with increasingly large amounts of data and varied sources, which make data ingestion a more complex process.  It is important not only be prepared for the present state of data ingestion, but to look ahead as things change quickly. Here are a few best practices to consider as you reflect on your data ingestion strategy:   1. Determine whether you need batch streaming, real time streaming, or both. Data can be ingested via batch data source, streaming data source, or a hybrid. Some businesses have to remain on premise due to regulatory or business requirements, while others are able to take advantage of the cloud. You can learn more about how to ingest data with Oracle Data Integrator and Oracle GoldenGate here.   2. Before you begin the ingest process, determine where your data will go and set up the proper hubs. Data lakes, data marts, and data warehouses are all separate entities, which people can sometimes confuse for one another. Before you begin to ingest your data into one or more of these spaces, consider the business problem you are trying to address. Based on your purpose, decide where the data will best live. To clear up any confusion: data warehouses ingest data via ETL process and are given a pre-assigned schema to handle analytical processing. Data marts are similar to warehouses, but they handle more restricted, specific groupings of data – data marts are sometimes used to process smaller subsets of data that complement what is stored in a warehouse. Finally, data lakes are optimal for massive storage and analysis of both structured and unstructured data. Data lakes offer high availability, low costs, and increased flexibility ideal for managing big data.   3. Be patient and manage your time accordingly.  According to a study cited in Forbes, data scientists spend about 80% of their time preparing and managing data before analysis. This is time that would otherwise be invested analyzing data; however, it is critical that when data is ingested, it is clean and prepared for analysis. IDC predicts that, by 2020, spending on data preparation tools will grow 2.5 times faster than tradition IT tools for similar purposes – these tools will be built with the intention of simplifying and speeding up the data ingestion process.   4. Get on board with machine learning. Not too long ago, data ingestion could be performed manually. One person could define a global schema and simply assign a programmer to each data source. The programmers could then map and cleanse the data into the global schema. Now, the volume of data and the number of sources are too large for programmers to follow the same plan. Artificial Intelligence (AI) is quickly changing the way in which we process large amounts of information. It is helping us work more efficiently and is identifying patterns, definitions, and grouping that are often lost to human error and oversight.   5. Data governance is key.   Data quality and security are extremely important for ensuring the accuracy and reliability of the insights and analysis the data reveals. It is one thing to initially cleanse data, but it takes a continued effort to maintain it. A data steward is responsible for maintaining the quality of each data source after the initial ingestion point. A data steward determines the schema and cleansing rule, decides which data should be injected into each data source, and manages the cleansing of dirty data. Of course, data governance is not limited to clean data – it also includes security and regulation compliance. Keep in mind that data is cleansed and governed after ingestion to preserve its original context and to allow for further discovery opportunities. It is, however, important to plan the governance process ahead of time, so it is ready to manage data after ingestion.   Conclusion Data ingestion is the first step in the data pipeline and it is hugely important. Businesses rely on good data to help them make smarter decisions, so it is important to take good care of your data from Step 1. Start by identifying your business needs and the resources you will need to meet them. As you go through the data ingestion process, be prepared for setbacks and always plan ahead. For more information, check out Oracle Data Integration Platform Cloud (DIPC) for a comprehensive data integration platform. DIPC provides all the functional components necessary to fulfill raw ingestion in batch or real-time, as well as the secondary phase of processing and operationalizing transformations and data quality routines.    

Data is the starting point of every smart business decision. Virtually every improvement a business makes these days – anything from sales strategies to IT processes to web content and so on –...

Big Data Success with Oracle Data Integration

Everyday Data Integration It is not every day that the average citizen wakes up thinking “Let me integrate some data today,” but that is precisely what they end up doing in many ways. From waking up to catching a favorite morning program on cable TV, to grabbing that cup of coffee from a major coffee retailer around the block, to binge-watching a favorite series on one of the leading subscription entertainment services, we come in daily contact with the business end of data integration. Of course, there is a nice human interface that contextualizes and enhances our experiences delivered through our mobile phones and web browsers on-top of the data that we interact with. Insightful businesses and business leaders want to provide the best services and they also look to delight and enhance customer experiences by providing personalized content and offerings. For this, deep insights and a complete understanding of customers is required. Data provides the raw information for these insights. Data also serves as the key ingredient, which when combined with other related data, bubbles up key recommendations that go into many every day and strategic business decisions that allow companies to differentiate themselves by providing customized and excellent capabilities. Many times, businesses tend to overlook the amount of planning and foundational work that goes into constructing the right data integration foundation for many transformation practices. In this series of blogs, I hope to bring out the role of data integration through various stories where Oracle Data Integration plays a critical role. LinkedIn LinkedIn is a great example of how Oracle Data Integration works behind the scenes to keep user experiences seamless. From ensuring that user profile data is accessible across the globe to making sure that user updates flow through instantaneously at the click of a refresh button, Oracle Data Integration underpins a well thought out architecture for one of the largest networking platforms. Here is a quick overview of how LinkedIn uses Oracle Data Integration, specifically Oracle GoldenGate for Big Data, to pass on the benefits of real-time updates and synchronization to its user community. A Deeper Dive On How LinkedIn Achieved Real Time Glory Architecture The primary goal was to ensure that data is efficiently moved across a number of online data stores (Oracle and MySQL) to downstream data stores, eliminating bulky data dumps, multiple hops, and standardizing the data formats. Oracle GoldenGate, the real-time replication engine that captures data changes and streams them to any required destination instantaneously, is at the core of this implementation. Oracle GoldenGate comes with off-the-shelf integration with many sources and targets making it easier to implement and integrate with existing systems that were being used. This allows greater compatibility with Oracle and MySQL databases also eliminating the multiple hops that the data used to go through to reach its final data store. Another critical criterion to select Oracle GoldenGate was the low impact on the source data base when capturing data to be streamed. Oracle GoldenGate for Big Data has a very light footprint and has minimal impact on business-critical source systems, an important consideration when fiddling around with carefully tuned applications and databases. Big Data Considerations – Kafka\HDFS Big Data platforms provide a cost-effective alternative to specialized hardware for both storage and computing for many scenarios. While data warehouses and high-speed analytics still benefit from optimized hardware, big data has gained tremendous viability for deep data storage used for machine learning and artificial intelligence requirements. Oracle Data Integration has always recognized this need for heterogeneity, a factor that played into the decision to use Oracle GoldenGate for Big Data for this project. Oracle GoldenGate for Big Data has handlers (a handler being a prebuilt bit of technology to integrate with specific systems) for many big data standards and technologies, among others, Kafka and Hadoop Distributed File System (HDFS). Both came in useful in pilot stages when determining which Big Data technology to adopt. Take a look at this datasheet for a more exhaustive capability list for Oracle GoldenGate for Big Data. But in short, this is how it can look: The Oracle Data Integration Platform LinkedIn is one of many customers who is innovating in their core businesses with cutting edge technology from Oracle. Oracle Data Integration, in turn, is pushing the boundaries of data integration to enable customers solve data challenges and turn them into opportunities for excellence. Oracle Data Integration’s products and services are evolving into a comprehensive and unified cloud service that brings together all the capabilities required for a data integration solution. The new platform, Oracle Data Integration Platform Cloud (DIPC), combines rich capabilities, a wide breadth of features, persona-based user experience, simple pricing, and packaging to make our customers’ data integration journey easier.   Learn more about Oracle Data Integration Platform Cloud here. Learn more about Oracle Data Integration here. Learn more about Oracle GoldenGate for Big Data here.

Everyday Data Integration It is not every day that the average citizen wakes up thinking “Let me integrate some data today,” but that is precisely what they end up doing in many ways. From waking up to...

Oracle Data Integrator (ODI) to Extract Data from Fusion Application (Oracle ERP Cloud)

Guest Author:  Srinidhi Koushik, Senior Sales Consultant at Oracle Oracle Data Integrator (ODI) to extract data from Fusion Application (Oracle ERP Cloud) In continuation to the previous blog wherein we discussed about loading data onto ERP Cloud, we will now discuss how to extract data from ERP Cloud using Oracle Data Integrator (ODI). ODI is available on-premises as well as in the Oracle Cloud with Data Integration Platform Cloud and ODI Cloud Service. Overview of the integration process In this blog, we will discuss the steps involved in downloading data from ERP Cloud. The steps involved are listed below The construct of data extraction is executed via the Web service call which in turn invokes the BI publisher report associated with the web service call The response of the Web Service call is a BASE64 construct string against “ReportBytes” tag Command: sed '/<ns2:reportBytes>/!d;s//\n/;s/[^\n]*\n//;:a;$!{/<\/ns2:reportBytes>/!N;//!ba};y/\n/ /;s/<\/ns2:reportBytes>/\n/;P;D' /home/oracle/Elogs/Report2_Resp.xml | tr ' ' '\n' c. The Java program is used to decode the BASE64 onto required format of file Java Construct of Program: import java.io.ByteArrayOutputStream; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.FileReader; import java.io.IOException; import java.io.InputStream; import java.io.BufferedReader; import java.io.FileReader;   import org.apache.commons.codec.binary.Base64;   public class DecryptBaseFile {     public DecryptBaseFile() {         super();     }       public static void main(String args[]) {         BufferedReader br = null;         FileReader fr = null;           try {             fr = new FileReader("/home/oracle/Elogs/ReportBytes.txt");             br = new BufferedReader(fr);               StringBuffer fileData = new StringBuffer();             String sCurrentLine;               while ((sCurrentLine = br.readLine()) != null) {                 fileData.append(sCurrentLine);             }             System.out.println("Reading data done from file /home/oracle/Elogs/ReportBytes.txt");               // Run the api to perform the decoding             byte[] rbytes = Base64.decodeBase64(fileData.toString().getBytes());               FileOutputStream os = new FileOutputStream("/home/oracle/TgtFiles/Customer.csv");             os.write(rbytes);             os.close();             System.out.println("File Customer.csv created successfully at /home/oracle/TgtFiles");         } catch (Exception e) {             e.printStackTrace();             System.out.println("Error:Kindly contact admin");             System.out.println(e.getMessage());         }     } } d. The created file is then copied onto source directory e. The file is then loaded onto necessary tables The ‘OdiInvokeWebService’ construct provided below Please note: OWSM rule that needs to be selected is oracle/wss_username_token_over_ssl_client_policy Requirement for this to work is to use ODI Enterprise configured with an Agent deployed in WebLogic Server The Topology associated with the report generation can be found below Inputs Sought by Web service: Conclusion: The features available in ODI make it relatively simple to generate a file, compress / zip the file and call specific Java / Linux scripts / functions and invoking necessary web services on ERP cloud to download the necessary data onto any specific format like csv, html, txt, excel, etc.

Guest Author:  Srinidhi Koushik, Senior Sales Consultant at Oracle Oracle Data Integrator (ODI) to extract data from Fusion Application (Oracle ERP Cloud) In continuation to the previous blog wherein we...

Data Integration

Oracle Data Integrator (ODI) to Load into Fusion Applications (Oracle ERP Cloud)

Guest Author:  Srinidhi Koushik, Senior Sales Consultant at Oracle Oracle Data Integrator (ODI) is the best tool to extract and load onto Oracle ERP Cloud given its dexterity to work with complex transformation as well as support for heterogenous source / target format. ODI is available on-premises as well as in the Oracle Cloud with Data Integration Platform Cloud and ODI Cloud Service (ODI-CS). In this article we will walk you through how to upload data into Oracle ERP Cloud Overview of the integration process In this series of blogs we discuss the two facets of Integration: Extraction and Loading. In this specific article we will concentrate on Loading data onto ERP Cloud using the following process: a) Create a data file based on the standard Template (list of Mandatory columns)     required by the table within Oracle ERP cloud         List of attributes or mandatory columns needs to be derived and a template      created. List of columns can be seen from the link below         https://docs.oracle.com/en/cloud/saas/financials/r13-update17d/oefbf/Customer-Import-30693307-fbdi-9.html         The file name in this case containing customer data has to follow a given       naming convention as per above documentation eg: HzImpPartiesT.csv b) Zip the content of the file     The above file created as per ELT / ETL process needs to be zipped as per     upload requirement as well as to compress the data     OdiZip can be used alternatively a script can also be written to zip the file c) Convert the zip file onto BASE64 string using java program or alternate     programs    Java Program construct: utilEncodeBase.java file content import java.io.ByteArrayOutputStream; import java.io.File; import java.io.FileInputStream; import java.io.FileReader; import java.io.IOException; import java.io.InputStream; import org.apache.commons.codec.binary.Base64;   public class utilEncodeBase {     public utilEncodeBase() { super(); }   public static void main(String[] a) throws Exception {   // Enter the filename as input     File br = new File("/home/oracle/movie/moviework/odi/CustomerImport.zip"); // Convert the file into Byte byte[] bytes = loadFile(br);   // Call the api for Base64 encoding byte[] encoded = Base64.encodeBase64(bytes); String encStr = new String(encoded); // Print the file System.out.println(encStr);   }                      private static byte[] getByteArray(String fileName) {                     File file = new File(fileName); FileInputStream is = null; ByteArrayOutputStream buffer = new ByteArrayOutputStream(); int nRead; byte[] data = new byte[16384]; try { is = new FileInputStream(file); while ((nRead = is.read(data, 0, data.length)) != -1) {     buffer.write(data, 0, nRead); } buffer.flush(); } catch (IOException e) { System.out.println("In getByteArray:IO Exception"); e.printStackTrace(); } return buffer.toByteArray(); }                      private static byte[] loadFile(File file) throws IOException { InputStream is = new FileInputStream(file);   long length = file.length(); if (length > Integer.MAX_VALUE) { // File is too large } byte[] bytes = new byte[(int)length];   int offset = 0; int numRead = 0; while (offset < bytes.length && (numRead = is.read(bytes, offset, bytes.length - offset)) >= 0) { offset += numRead; } if (offset < bytes.length) { throw new IOException("Could not completely read file " + file.getName()); } is.close(); return bytes; } } d) Create a XML file to be passed via the Webservices to load the file. We have     java program to load the BASE64 string to XML and in turn use this XML as     the request file. The below program helps generates a XML with input     string (BASE64) Java Program construct: genxml.java file content import java.io.BufferedReader; import java.io.FileOutputStream; import java.io.FileReader; import org.jdom.Document; import org.jdom.Element; import org.jdom.Namespace; import org.jdom.output.Format; import org.jdom.output.XMLOutputter; public class GenerateXML {   public GenerateXML() {}     public static void main(String[] paramArrayOfString)   {     try     {       BufferedReader localBufferedReader = null;       FileReader localFileReader = null;               localFileReader = new FileReader("/home/oracle/Elogs/Conv_Out.txt");       localBufferedReader = new BufferedReader(localFileReader);             StringBuffer localStringBuffer = new StringBuffer();             String str;       while ((str = localBufferedReader.readLine()) != null) {         localStringBuffer.append(str);       }         Document localDocument = new Document();         localDocument.setRootElement(new Element("Envelope", "soapenv", "http://schemas.xmlsoap.org/soap/envelope/"));       localDocument.getRootElement().addNamespaceDeclaration(Namespace.getNamespace("typ", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/types/"));       localDocument.getRootElement().addNamespaceDeclaration(Namespace.getNamespace("erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/"));        Element localElement1 = new Element("Header", "soapenv", "http://schemas.xmlsoap.org/soap/envelope/");       localDocument.getRootElement().addContent(localElement1);          Element localElement2 = new Element("Body", "soapenv", "http://schemas.xmlsoap.org/soap/envelope/");       Element localElement3 = new Element("uploadFileToUcm", "typ", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/types/");       Element localElement4 = new Element("document", "typ", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/types/");       Element localElement5 = new Element("Content", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement5.addContent(localStringBuffer.toString());       localElement4.addContent(localElement5);        Element localElement6 = new Element("FileName", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement4.addContent(localElement6);       Element localElement7 = new Element("ContentType", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement4.addContent(localElement7);         Element localElement8 = new Element("DocumentTitle", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement4.addContent(localElement8);       Element localElement9 = new Element("DocumentAuthor", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement4.addContent(localElement9);       Element localElement10 = new Element("DocumentSecurityGroup", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement4.addContent(localElement10);       Element localElement11 = new Element("DocumentAccount", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement4.addContent(localElement11);         Element localElement12 = new Element("DocumentName", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement4.addContent(localElement12);        Element localElement13 = new Element("DocumentId", "erp", "http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/");       localElement4.addContent(localElement13);       localElement3.addContent(localElement4);       localElement2.addContent(localElement3);       localDocument.getRootElement().addContent(localElement2);          XMLOutputter localXMLOutputter = new XMLOutputter(Format.getPrettyFormat());       localXMLOutputter.output(localDocument, new FileOutputStream("/home/oracle/TgtFiles/uploadfiletoucm_payload.xml"));     }     catch (Exception localException) {       localException.printStackTrace();     }   } }  e) Upload the file to UCM-Cloud using the appropriate web service call with      BASE64 as an input value against "Content" tag on XML Request file: Response File: (need to capture value against tag “result” f) Extract the value of result and capture the result onto a Variable grep -oE '<result xmlns="http://xmlns.oracle.com/apps/financials/commonModules/shared/model/erpIntegrationService/types/">[^<]*</result>' /home/oracle/Elogs/Con_Resp.xml | awk -F"[<>]" '{print $3}' g) Assign this value onto a Variable to be used for subsequent webservice call h) Invoke ERP-Cloud to trigger the import process, using the appropriate web     service call. Here is an example of this process implemented in an ODI Package: The ‘OdiInvokeWebService’ construct and the topology created for this requirement provided below Please note: OWSM rule that needs to be selected is oracle/wss_username_token_over_ssl_client_policy Requirement for this to work is to use ODI Enterprise configured with an Agent deployed in WebLogic Server Inputs Sought by Web service: Screenshot from ERP Cloud of loaded customers and statistics: Customers loaded in ERP Cloud from ODI: Conclusion: The features available in ODI make it relatively easy to generate a file, compress / zip the file and call specific Java / Linux scripts / functions and invoking necessary web services on ERP Cloud to aid uploading the necessary files onto specific ERP Cloud directory.

Guest Author:  Srinidhi Koushik, Senior Sales Consultant at Oracle Oracle Data Integrator (ODI) is the best tool to extract and load onto Oracle ERP Cloud given its dexterity to work with...

Data Integration

DI Connection: Where Machine Learning Meets Data Integration

Much of the data we manage day-to-day lives in silos and, while the data within each silo can be analyzed on its own, this produces limited value for businesses. The deeper value becomes apparent when the disparate data is connected and transformed to produce patterns, insights, and predictions. This undertaking requires a significant amount of time and labor only to be susceptible to human error. As businesses are faced with analyzing big data from heterogeneous sources as quickly as possible, data integration will increasingly look to automation and machine learning for the heavy lifting. Businesses are pulling data from CRM systems, social media, apps, and a swarm of other possible sources and this data is often unstructured. With so much information available and constantly updated in real time, it is impossible to keep up with all the data through human power alone. Therefore, machine learning becomes necessary when handling data across disparate sources. Artificial Intelligence continuously develops new and improved algorithms as it is fed with more data, becoming smarter and more agile in finding connections between and among various data points. Machine learning is on track to become a driving force within data integration systems. Here are a few ways in which machine learning is changing the game: Decision-making: AI-powered predictive analytics find insight in data beyond human capability. One of the greatest strengths an AI possesses is the ability to connect data points from heterogeneous sources and identify previously unrecognized connections and patterns. This creates a new level of transparency when making decisions and empowers businesses and leaders to make more thoughtful, better-informed choices to increase profits, to improve internal efficiency, to create more targeted brand messaging, and so on. Productivity: Machine learning helps automate data-related tasks that are too large or complex for manual processing or simply too tedious. For example, AI can categorize data and identify patterns, screen for data quality, and suggest next best actions. These capabilities save both time and money by reducing human labor and potential errors that would require additional time-consuming work to repair. Changing Workloads: AI cloud-based workloads are quickly growing and offer the benefits of automated scalability and elasticity as well as the speed and ability to integrate with a variety of other tools. Workloads change through the day, month, and year, and businesses need the flexibility and high availability to handle those changes. The underpinnings of machine learning are already visible in Oracle’s on premise Data Integration portfolio and Data Integration Platform Cloud offerings. Currently, this is most obvious when it comes to profiling in the catalog, which allows users to get more information about connections and metadata before actually beginning the integration process. Another project in progress is a tool that can take in unstructured data and normalize it to then create data sets within a data lake. In the future, this will lead to better predicting how we want to use data – whether it is better for querying or processing and how to move it, whether in real time or in batch. Long term, we plan to integrate machine learning and AI on multiple levels of our data integration offerings. A couple areas in particular will include 1) AI-fueled monitoring to predict outages and lags and 2) building out an intelligent designer, which will eventually make data mappings almost entirely automatic and offer relationship and transformation recommendations. Machine learning will surely become an integral part of every strong, adaptive data integration solution in the near future.  Ultimately, the role of artificial intelligence in data integration is to increase automation and to strengthen intelligence via machine learning in order to decrease integration costs, human labor, and human error, while producing valuable business intelligence that leads to more strategic decision-making, better productivity, and more accurate, targeted marketing, among other possible benefits.                                                                                                                                                  

Much of the data we manage day-to-day lives in silos and, while the data within each silo can be analyzed on its own, this produces limited value for businesses. The deeper value becomes apparent when...

GoldenGate Solutions and News

Oracle GoldenGate Veridata 12.2+ new enhancements

In series of blogs around Oracle GoldenGate Foundation Suite (OGFS) products, I mentioned the new features for OGG Monitor in my previous blog. Today, I am pleased to announce that we have released the latest OGG Veridata Bundle patch(12.2.1.2.171215) end of December. In this Veridata Bundle patch, we have provided support for Hive (Big data), and automatic key column mapping. Let me first write about automatic key column mapping. In the absence of unique identifiers, such as the primary keys and unique indexes, which can uniquely identify each row, you need to manually define any column or columns to be used as a unique identifier. If you have hundreds of such tables, then it is an excessive work to select PKey for each table in the user interface (UI). However, there was a workaround to use the Veridata GoldenGate Parameter Processing (VGPP) and create the compare pair. It was still requires some work to be done from your side. Now with this new bundle patch, Veridata will select key columns automatically for you based on specific inputs provided while creating compare pairs or connections. If you have enabled automatic key column mapping at the Connection Level, then any group using this Connection has the feature enabled. You can override automatic key column mapping at the Compare Pair Generation stage. The connection level has mainly two new options. Use Source or Target Columns as Key Columns When Generating Compare Pairs: If the source table has primary keys or unique index defined, but the target does not, then Oracle GoldenGate Veridata uses the same columns (as the source) as unique identifiers for the target. Similarly, if the source does not have the primary key or index, but the target has them, then key columns of target are used for source. If either (source or target) of the columns of the Primary Key or index is not present on the target side, then the primary key or index is not considered. Use All Columns as Key Columns When Generating Compare Pairs: Enables Automatic mapping for the source and target connections. If this option is disabled either at the source or the target connection, then the automatic mapping is also disabled for that group. If you enable this option to map all columns from source and target, then the mapping is considered only when both source and target table do not have primary or unique keys. If you do not want to specify "automatic key selection" options at the connection level, you can do so while configuring compare pair as well. The Manual and Pattern compare pair configuration has following new options:- Use Source Key Columns for Target: Enables automatic mapping for the target table. If you disable the option for the target side, and if there is no primary key or unique index on the target side and the source has primary key or unique index, then only the mapping of columns happen but, automatic mapping does not happen on the target table. Use Target Key Columns for Source: Enables automatic mapping for the source table. If you disable the option for Source side, and if there is no primary key or unique index on the source side and target has primary key or unique index, then only the mapping of columns happen but, automatic mapping does not happen on the source table. Use All Columns as Key Columns When Generating Compare Pairs: Enables automatic mapping for all columns. Ensure that this option is selected both at the source as well as the target groups. If the option is disabled either at the source or the target levels, then the automatic mapping is disabled for the corresponding groups. Once you have provided your inputs and compare pair is created, you can see what Veridata has selected for each column in "Key Mapping Method" as mentioned below in the figure.  The other important feature was released in the bundle patch is supporting Big Data Hive. We know that you must be having your data replicated in big data targets. However, how will you make sure that the replicated data is correct? Veridata has always provided validation support for Databases. Now we have started and provided support for our first Big data target Hive. You can compare data between your source relational databases (Oracle, MSSQL, DB2 LUW, Sybase, Informix, etc.) and Big Data hive (2.1.1+). Even you can do Hive to Hive comparison too. The Hive comparison need not require any different options to be selected; you can make the Hive comparison just like you do Oracle to Oracle comparison. You would need to create the Hive connection (for your source or target) and create groups which will contain compare pairs from the source database and Hive database. Once all the compare pairs and options for primary key selection is selected, you can run the job. The Hive feature supports Delta comparison and raw partition. However, it does not support Hive authentication and Repair feature as of now.   Let me know if you have any questions.    

In series of blogs around Oracle GoldenGate Foundation Suite (OGFS) products, I mentioned the new features for OGG Monitor in my previous blog. Today, I am pleased to announce that we have released...

Data Integration

Data Integration 2017 Wrap-Up

We are wrapping up 2017 with a summary of all the latest announcements, product features, and happenings in our Data Integration space. I sat down with one of Oracle’s best Product Management experts in the DI space, Julien Testut, Senior Principal Product Manager, to get his insight into the latest happenings. Q: Can you summarize the major accomplishments achieved in these past few months? Without a doubt, it is the recent launch of the Oracle Data Integration Platform Cloud (DIPC). This is extremely exciting as it unifies everything we have been doing for years in the Oracle Data Integration team into a single unified Cloud offering: data movement, data preparation, data transformation and data governance. We are off to a great start and are already pushing our second update this month with release 17.4.5. It includes a brand new user experience with the DIPC Console, a central data catalog with powerful search capabilities, an innovative way to solve data synchronization problems through a powerful yet easy to use Task as well as Remote Agents that can be installed anywhere and give us access to any data. In addition, DIPC comes with full access to Enterprise Data Quality, Data Integrator, and GoldenGate so it can truly address any data integration requirement. Q: Were there any great customer stories that you would like to share with our readers? I had the pleasure to get on stage with ARRIS at OpenWorld this year and I think their story is a great example of what innovative customers can do with our Oracle Cloud technologies. ARRIS is a world leader in video and broadband technology and they have started to use our Oracle Cloud Applications and Platform including Oracle Data Integrator Cloud Service or Oracle Integration Cloud Service to provide a unified, modern business platform, improve agility and reduce costs. ARRIS was able to go from Development to Production in a matter of weeks with Data Integrator Cloud Service and the whole project went smoothly. I am very excited about our partnership with ARRIS and how we can help them go through their digital transformation now and in the future. Q:  What are some key data integration challenges that companies should be thinking about today? One key challenge is that data is now everywhere and how companies can harness value from that data is critical to their success. The great news is that most companies have realized that data is a core asset but we also know that creating value from data is hard. Not only is there a proliferation of data as I mentioned but it also now comes in different shapes and formats and needs to be processed in more flexible ways than ever before. This will keep data integration professionals busy for years to come! This requires using solutions that can deal with any data format and shape, can help integrate it with newer emerging technologies in batch or real-time as well as help govern it. Another critical challenge is how to enable business users to access the data they need in the shape they want and when they require it. As you know, the business demands faster data access in order to derive more value from it. This requires providing self-service data access to anyone who needs to consume data as well as more agile and flexible ways to process that data. Obviously, this raises questions around data governance as you still need to know who did what and when with the data and be able to report on it when required. Finally, and I think it is related to the challenges mentioned above, companies have to start thinking and implementing more automated data management processes using machine learning techniques. Since there is so much data out there and because data integration is so complex, companies will have to automate as much activities as possible in order to keep up. There are many data processing and data ingestions areas that require doing the same tasks repeatedly. Companies can benefit tremendously from some automation and recommendations in order to make their users even more productive. Obviously, we aim at helping companies tackle these challenges with our Data Integration Platform Cloud offering. You can watch this video for an overview tour of DIPC: Q: What else should people know about Oracle’s Data Integration offerings? There would be so much to talk about but let me point out a couple of things. First, Data Integration at Oracle is unique. For example, we were the first vendor to provide enterprise-class data replication capabilities from relational systems into Big Data with GoldenGate for Big Data. Oracle Data Integration is also highly differentiated we have been providing best-of-breed pushdown data processing capabilities with Oracle Data Integrator before it became mainstream at a time when most vendors still relied on separate hardware to handle transformations. Finally, Oracle delivers truly innovative functionality in that space. Data Integration Platform Cloud is a great example of that as no other vendor can provide the same level of functionality into a single offering and this is only the beginning of the DIPC journey for us! Q: What can our customers look forward to in the data integration space with Oracle in 2018? We have a very exciting line up in 2018 with several new releases coming up both in the Cloud and on-premises. Customers can expect us to accelerate the pace of innovation throughout our offerings and to continue to deliver the functionality they need to design their data-driven solutions and solve their challenges. You should also see many synergies between the various components of the Oracle Cloud platform. This is going to be critical for customers as they adopt and standardize on our Cloud Platform and want, for example, to reuse Data Integration processes in their Analytics or Application Integration solutions. Stay tuned for more announcements as we move forward! To learn more about Oracle’s Data Integration Platform Cloud, watch the webcast ‘Supercharge Your Data Integration.’ Happy Holidays from all of us at Oracle!

We are wrapping up 2017 with a summary of all the latest announcements, product features, and happenings in our Data Integration space. I sat down with one of Oracle’s best Product Management experts...

Data Integration Platform Cloud (DIPC) 17.4.5 is available!

Data Integration Platform Cloud (DIPC) was released about a month ago (Unveiling Data Integration Platform Cloud) … and already, we have more to tell you about! Data Integration Platform Cloud (DIPC) 17.4.5 is now available! Here are some of the highlights: DIPC boasts a new user experience, including a new landing page.  You will find shortcuts guiding you towards typical activities such as creating new Connections, new Tasks or browsing the Data Catalog.  The DIPC Catalog is now available, providing access to a unified metadata repository.  You will be able to browse metadata artifacts: Connections, Data Entities, and Tasks.  You will also have Advanced Search capabilities with inline results and property search, as well as dedicated pages for each object with additional information including a data viewer and profiling metrics for Data Entities. The first elevated task, Synchronize Data, has arrived.  Let DIPC take you through how easy it is to synchronize data between a source and a target schema!  The process is easy to configure and run.  This Task will do an initial load using Oracle Data Integrator (ODI).  Then, changed data is captured and delivered using Oracle GoldenGate (OGG).  Synchronizing OGG and ODI together ensures the best performance and zero data loss! Monitoring is key, right?  DIPC includes powerful monitoring capabilities where every Job execution is displayed along with the corresponding metrics and errors.  You are able to filter and search job instances, or drill-down into detailed runtime information.  You can also set up Policies and receive Notifications. This release of DIPC also introduces the Remote Agent functionality. The Remote Agents can be installed anywhere and help DIPC move data from on-premises systems to the cloud and vice-versa. Are you doing on-premises to on-premises work?  DIPC can still help!  Remote Agents give you the ability to create custom replication processes that will move data between two on-premises systems without routing any data to the cloud! Want to learn more?  Visit the DIPC site and view the Webcast: Supercharge Your Data Integration.

Data Integration Platform Cloud (DIPC) was released about a month ago (Unveiling Data Integration Platform Cloud) … and already, we have more to tell you about! Data Integration Platform Cloud (DIPC)...

Data Integration

Data Integration Corner: Catching up with Chai Pydimukkala, Sr. Director, Product Management

With Oracle’s new Data Integration Platform Cloud (DIPC), our customers have access to a truly unified, streamlined way to manage and transform their data to maximize its business value. For the final chapter in the Data Integration Corner blog series, I sat down with Chai Pydimukkala, Senior Director of Product Management, to get a deeper understanding of the value of data and the unique offerings of DIPC.   Q: Hi Chai, to start, please tell me a bit about yourself and what you do here at Oracle. A: I joined Oracle almost 4 years ago but I have been in the data integration and data management space for around 17 years now. At Oracle, I focus on data replication and GoldenGate technologies on the product management side, and I also oversee and help with the overall product strategy for data integration as a whole. Q: How is DIPC changing how companies approach data integration? A: Data integration is not a new area. We have seen a lot of companies come and go and most of the technologies that have been built in this space have been trying to address one problem: just ETL, just replication, or just data quality, for example. All of these companies were trying to solve data integration problems in a siloed fashion. Oracle has all the individual tools, as well, but we recognized the unique opportunity to combine all our data integration features and functionalities. We made sure to keep the individual tools intact for our tens of thousands of existing customers. But then, we were able to distill and make all these same functionalities available to users in a fluid, cohesive way without making them choose among a set of separate tools. With this platform, they get extensive capabilities related to data integration and it helps customers adopt a streamlined way to manage their data and to meet certain needs before they may be aware they even need certain functionalities. There is no other company that can provide a comprehensive data integration cloud suite available in the market today. We look at metadata management with a task-driven approach: if you look at enterprises, they start with a business problem. For example, “I want to migrate my business data from on premise to the cloud. How can I do that?” or “I want to create a reporting system in the cloud.” In the operational world, you have to break down the available tools and do a lot of granular level work to figure out how to get what you need. We are making it easy for our customers: you come into our system, we walk you through a few user-friendly steps so you can say “I want to go from Point A to Point B” and then you implement the task without all the messy work in between. When this happens, you can go back and audit your data to understand on a more broken-down level what happened to your data. That’s really the unique value proposition of DIPC. Q: How are data integration and app integration different? A: Integration is a problem everyone faces. At the core, integration is nothing but exchanging information between various systems. Of course, that information comes in many different formats and from different places and that is where it becomes difficult. In data integration, we are not really looking at the business context – we don’t differentiate between a customer, an employee, or a service ticket, for example. Everything is done via tables. We are not doing even-space integration. For example, say you added a new employee and their joining your company is marked as an event in your HR system. As soon as the employee is added, you have to make sure his information is uploaded. Let’s say you are doing your payroll through a third party and need to alert that third party. That information exchange is application integration. Whereas in a data integration issue, let’s say you are looking to find patterns in your employees’ salaries and want to find the average, the maximum, and the minimum you are paying. You end up pulling all the data from the HR system and creating a report. This is an example of a data integration problem. Data integrations happens when you have large amounts of data you want to transform in order to find new insights. Q: Why is data so important? A: Customers have realized that data is the biggest asset companies have. For example, I used to receive a huge booklet from one retailer years ago – it covered all of their products, whether they were relevant to me or not, and the booklets were costly to produce and to distribute. They did this because they did not have enough data about their customers, so their strategy was to cover all the bases as a precaution. Once they realized the negative impacts of this strategy, they built a Customer 360 system using data from social media, directly from customer transactions, and from other sources. This allowed them to understand what each customer was looking for and to then create very specific, targeted advertisements. The booklets became much more concise and relevant to my purchase history and were supplemented with targeted digital ads. Targeted marketing campaigns have exploded the last few years as companies have come to realize the time and resources the can save and the revenue boost they can gain by treating customers as individuals. None of that would be possible without data. The only way you can grow your company’s revenue is if you know your customers completely and you know where they will invest their money.     As the holidays and New Year approach, the “Data Integration Corner” series comes to a close with this blog. It has been a pleasure to pick the brains of several members of Oracle’s Data Integration product management team. I encourage you to go back and catch up on the previous posts in this series with Nishit Rao, Denis Gray, and Sandrine Riley.   To learn more about DIPC, take a look at a recent article from DBTA:  Oracle Announces Comprehensive Data Integration Platform Cloud and check out the eBook & data sheet.  

With Oracle’s new Data Integration Platform Cloud (DIPC), our customers have access to a truly unified, streamlined way to manage and transform their data to maximize its business value. For the final...

GoldenGate Solutions and News

Oracle GoldenGate Monitor 12.2 new enhancements

Data Integration needs are growing day by day, and especially your replication needs are also increasing as your business evolves. Hence we at Oracle are strived to provide the new enhancements for the products around Oracle GoldenGate to make your experience better. I am talking about Oracle GoldenGate Foundation Suite (OGFS) which is a broader umbrella covers Oracle GoldenGate Studio and Oracle GoldenGate Management Pack (Veridata, GoldenGate Monitor, Director, REST-based Interface and OEM Plug-In). In series of blogs, I will be updating you about the new features that we have recently added for OGFS products. Let's start with the GoldenGate (OGG) Monitor in this blog. OGG Monitor Bundle Patch (12.2.1.2.171115) was released end of November. Mainly, we have provided three enhancements and few improvements around the product. When you are using Oracle Big Data Adapter, it requires you to provide the .properties configuration file for each Big data targets. These .properties files are needed to store the big data target related metadata information. Now with this release, you can edit, save the OGG for Big Data parameter file and .properties file from Monitor editor. All the configurations can be viewed/ modified/saved from Monitor centrally. The other enhancement was around improving the purging history table. GoldenGate Monitor stores historical data in the repository database, and you can use Oracle or SQL Server databases for the same. In this release, we improved the performance of the purging historical data for Oracle database repository. When your historical information grows further, purging and maintaining the massive amount of data was time-consuming. Every month, OGG Monitor will create the new partition and stores all the data into the new partition. When you purge the historical data, the partition will allow Monitor to purge the data faster. If you are an existing user and want to get the benefit of purging data improvement, you can partition your current history table by executing the oracle_partitioning.sql script after an upgrade. If you choose not to execute the oracle_partition.sql script after the upgrade, your repository table will not be changed, and you would continue running the repositories as you would earlier. Oracle GoldenGate has released the major release in last month, OGG 12.3. In this version, Oracle GoldenGate supports two modes, Classic and Microservices architecture. The Microservices architecture is a new age of Cloud ready, REST services-based architecture, empowers you to design, build, embed, and automate all aspects of your replication environment on premise and in the cloud. You can find more details around the OGG 12.3 from the blog written by my colleague Thomas Vengal. Release Announcement for Oracle GoldenGate 12.3. OGG Monitor certifies the OGG 12.3 Classic mode. Now you can monitor the OGG 12.3 Classic mode Instances from OGG Monitor. To Monitor the OGG 12.3 heterogeneous platforms using classic mode, please make sure that you have a more recent patch installed for OGG. In future articles, I will write about OGG Studio, OGG OEM PlugIn and Veridata.

Data Integration needs are growing day by day, and especially your replication needs are also increasing as your business evolves. Hence we at Oracle are strived to provide the new enhancements for...

Data Integration

Data Integration Corner: Catching up with Sandrine Riley, Principal Product Manager

In her role as Principal Product Manager in a team focused on data integration, Sandrine Riley has a unique, overarching perspective on the field. I recently spent some time with Sandrine learning about her role, how Oracle’s new Data Integration Platform Cloud fits into the current landscape, and how that landscape is evolving. Take a look and make sure to register for our webcast on December 7 for an in-depth look at the Data Integration Platform Cloud. Q: Hi Sandrine, tell me about yourself and your role here at Oracle. A: I am one of the product managers in the data integration area and I have a somewhat unique role where I don't necessarily interface with development on a day-to-day basis like many other product managers do. I work a lot more directly with our Customer Advisory Board and on many of our events that are data integration heavy, so I have the opportunity to really hear from our customers on a first-hand basis and understand their needs and experiences on a more personal level. Q:  What does a typical day as a data integration PM look like for you? A: It really varies because I work across a broad range of solutions including GoldenGate, Enterprise Data Quality, Enterprise Metadata Management, Oracle Data Integrator, and of course, Data Integration Platform Cloud. A lot of it has to do with interacting with customers - evangelical work where we are really trying to make sure our customers understand our new products and we understand their needs.  Q: What is one word that comes to mind for "data integration"? A: The first word that comes to mind is "plumbing." The importance of data integration is not always clear at face value but it's the foundation below entire enterprises. Q: What kind of customer is Data Integration Platform Cloud built for? A: What's really unique about DIPC is the fact that we're combining so many varying personas within an organization. It's the first time you'll see so many functionalities wrapped into one platform. Realistically, this addresses all customers because it is a need all customers have. If the data integration part of your business is not done right, there are many other sacrifices that are made at higher levels that become more obvious – that really ties back to my ‘plumbing’ comment. That's why it's so important to get data integration right, so that you can make valuable, insightful, and appropriate business decisions. In the end, DIPC is applicable across the board to all industries of all sizes, because data matters to everyone. Q: A lot of the data companies use resides outside the company. Why does that matter? A: We don't always have control over what kind of data we get. It may be in a format we don't understand or have not dealt with before. Being able to transform that into data we can process and incorporate it with other data is critical because data is one of our most valuable assets. If you don' have an appropriate data integration layer, it is really difficult to trust and value the insights that you think you are getting out of your data. Q: Where do you see the data integration field going in the next couple of years? A: I think we will continue to see more and more synergy in how we approach data integration. People need to do more with less and they are really interested in a better user experience. Our goal is to capture that and make a great user experience that is efficient and drives business value. As data sources become increasingly diverse, we are also seeing a need for simplicity in the ‘plumbing’ of our data integration tools. Streamlined processes and trustworthy data are and will continue to be two critical pillars of data integration. Data and its sources can be unpredictable and ever-changing, which is why the tools we use to make sense of it are so critical. I appreciated Sandrine’s perspective here – after working closely with so many of our customers, she as seen the direct impact tools like GoldenGate, Oracle Data Integrator, and others have had on our customers’ business value and she has also picked up on the even greater impact weaving these tools together has had. Research like this brought us to the Data Integration Platform Cloud, as we move forward in our goal of making data integration simple, reliable, and effective for all our customers.               

In her role as Principal Product Manager in a team focused on data integration, Sandrine Riley has a unique, overarching perspective on the field. I recently spent some time with Sandrine learning...

Data Integration

Walkthrough: Oracle Data Integration Platform Cloud Provisioning

We recently launched Oracle Data Integration Platform Cloud (DIPC), a brand new cloud-based platform for data transformation, integration, replication and governance. In case you missed it, you can get more information about it in Unveiling Data Integration Platform Cloud. In this article, I will focus on the provisioning process and walk you through how to provision a Data Integration Platform Cloud instance in the Oracle Cloud. First, you will need to access your Oracle Cloud Dashboard. You can do so by following the link you received after subscribing to the Oracle Cloud or you can go to https://cloud.oracle.com and Sign In from there. We will start with the creation of a new Database Cloud Service (DBCS) instance. This is the only pre-requisite before you can create a Data Integration Platform Cloud (DIPC) instance as DBCS is used as a repository for the DIPC components. From the Dashboard, select Create Instance: Then click on Create next to Database: Follow the instructions located in the DBCS documentation to create your Database Cloud Service instance: Creating a Customized Database Deployment NOTE: Make sure you create a Custom instance with Database Backups enabled (select ‘Both Cloud Storage and Local Storage’ under Backup Destination) when creating it to make sure it will be compatible with DIPC. When your Database Cloud Service (DBCS) instance is provisioned, you can move forward with the provisioning of your Data Integration Platform Cloud (DIPC) instance. Go back to the Dashboard, click on Create Instance, click on All Services and scroll down to find Data Integration Platform under Integration. Click on Create next to it. This will get you to the DIPC Service Console page: Click on Create Service to navigate to the provisioning screens. In the Service screen, enter a Service Name, the size of the cluster and the Service Edition. In this example, I have selected the Governance Edition that includes Data Governance capabilities in addition to everything included with DIPC Standard and Enterprise Editions.   Click Next to access the Details page. There you can: Select the Database Cloud Service (DBCS) instance you have previously created and enter its details under Database Configuration Specify your Storage Classic account and its details under Backup and Recovery Configuration Define the WebLogic Server Configuration for this DIPC instance. WebLogic is used to host components such as the DIPC Console, Enterprise Data Quality (EDQ) Director or Oracle Data Integrator (ODI) Console: When done, click Next to review the configuration summary: Finally click Create to start the DIPC instance creation. Next, we can track the progress of the Data Integration Platform Cloud (DIPC) instance creation in the DIPC Service Console. To go there we first need to show DIPC in the Oracle Cloud Dashboard. From the Dashboard, click on Customize Dashboard: Scroll down in the list and click Show under Data Integration Platform Cloud: Data Integration Platform Cloud (DIPC) will then start appearing on the Dashboard: Click the Data Integration Platform Cloud (DIPC) tile to see more details about your subscription and click Service Console to view your DIPC instances: You will then see your DIPC instance in the process of being created with the ‘Creating service…’ status: You can click on the instance name to get more details about it and the creation process (under In-Progress Operation Messages): When the provisioning process is over, the DIPC instance will show as ‘Ready’: That’s it! Congratulations! We now have a new Data Integration Platform Cloud (DIPC) instance to work with. You can get more information about it on the product page: Data Integration Platform. In future articles, we will cover how to start using the various components included with DIPC. Do not miss our webcast Supercharge Your Data Integration on December 7th to learn more about Oracle Data Integration Platform Cloud (DIPC).

We recently launched Oracle Data Integration Platform Cloud (DIPC), a brand new cloud-based platform for data transformation, integration, replication and governance. In case you missed it, you can get...

Data Integration

Data Integration Corner: Catching up with Denis Gray, Sr. Director, Product Management

Denis Gray, Senior Director of Data Integration Technology, has dedicated his career to the data integration space. I recently had the opportunity to sit down with Denis and pick his brain about all things data integration.   My conversation with Denis was eye opening. In just the last few years, customers’ data integration needs have experienced an overhaul. In particular, the need to process big data and merge data across disparate environments has meant customers need more powerful, more versatile, and faster tools. I was also reminded that we never truly know what our data is capable of telling us, we can never fully predict how it can move our businesses forward, until we are presented with a tool that can show us what it’s capable of.   Q: Hi Denis, tell me about your background with data integration and your role at Oracle. A:  I have been in data integration since 2000, when I began working with Hyperion Solutions. After Oracle acquired Hyperion in 2007, I started working with Jeff Pollock and found his team to be a really great fit for me.  Over the years, Oracle has added on capabilities thanks to the Golden Gate acquisition, various data quality acquisitions, and others. As we moved into the cloud, we saw an opportunity to bring all these capabilities together, which was the inception of the Data Integration Platform Cloud (DIPC).   Q: What are the stand-out features of DIPC? A:  In a nutshell, having a cloud platform for data integration allowed us to bring together best of breed data integration engines that we had for on-premise use and transform them for the cloud. So we looked at bulk-type transformations of data, real-time data integration, and being able to provide data governance, data lineage, and business impact along with metadata quality and enterprise management. Moving to the cloud allowed us to bring these products together and also to build additional functionality on top of that. Our core goal with DIPC and with all the technologies available before has been to help our customers derive more value from their data. We brought together this set of products to allow the DVA, the ETL developer, and the Data Steward to be more collaborative and to more efficiently meet their data integration needs. Q: What is the role of big data in data integration? A:  Big data has definitely brought integration tools a long way. The thing is, we can look at big data in relational databases without a platform but there are so many other elements that come into play like IoT, difference devices, and just generating massive amounts of data and being able to intertwine all that data. Some of the data our customers work with lives in relational databases but plenty of other data lives in big data environments, whether they’ve been pushed out to Spark or back to Hadoop itself, that’s where we’ve been seeing our customers struggle to connect data over the years.  All the new data integration tools available now are helping to bridge the gap between big data in disparate environments. Now we are empowering IT users and ETL developers to use machine learning to be able to use Spark libraries on their data sets or to use the massive parallelism within their big data and Hadoop environments to provide their transforms.   Q: How is Oracle addressing the diverse needs of its customers? A: We know that every Oracle customer out there has different types of relational databases as well as different types of sources and targets, so we made the decision early on that Oracle’s data integration solutions would support all of our customers’ needs.  When we first came out with Oracle Data Integrator (ODI) for example, we made sure it wasn’t just the best for Oracle, but it was the best for any type of relational database out there.  We are not locking any of our customers into a specific set of underlying technologies – we want to continue to be open and are committed to continuing to grow our ability to support as many other services as possible.  On top of that, we want to run not only in the cloud but also in hybrid mode so we can have our agents run on premise, in third-party data centers, and really with any type of source, any type of target, and any type of scale.   Q: Most companies are using one analytics software or another. Don’t those technologies already have elements of data integration built into them? A: Not really. If we were to look at the top analytics technologies, they have some limited capabilities that you may be ok with if you’re looking for high-level information once a month and are willing to wait 20 minutes for your report to run. If you want to end up with data in a format that many users can access that’s already been aggregated, transformed, and in a dimensional model, that’s where data integration tools need to come in.     To learn more about Data Integration, I invite you to join us on December 7 at 9am PST for a data integration deep dive webcast with Denis and Madhu Nair, Director of Marketing, Data Integration. Register now!

Denis Gray, Senior Director of Data Integration Technology, has dedicated his career to the data integration space. I recently had the opportunity to sit down with Denis and pick his brain about...