Friday Feb 10, 2012

Eliminating Batch Windows Using Real-Time Data Integration

When we invest in technology solutions we expect improvement in productivity, agility, performance and more. We don’t want to be limited by the technology we select. While data warehouses are designed to give us the freedom to access complete and reliable information, the underlying data integration architecture and the type of solution used can lead to significant constraints on how we manage our critical production systems.

With data warehousing solutions, one of the most common constraints is the time window available for batch extract processing on the source systems. The resource intensive extract process typically has to be done in off-business hours and restricts access to critical source systems.

A low-impact, real-time data integration solution can liberate your systems from batch windows. When the extract component uses a non-intrusive method, such as reading database transaction logs to capture only the changed data, it does not burden source systems. Hence, data extract can happen at any time of the day, and throughout the day, while all users are online.

SunGard is a great example for achieving a major transformation in data warehousing solutions by using log-based real-time data integration. The company removed batch processing timeline related constraint by using Oracle GoldenGate to capture all of the intraday changes that take place on the selected tables. As a result they reduced the nightly extract process from the Oracle E-Business Suite application by 9 hours. In addition, Oracle GoldenGate enables to feed EBusiness Suite data to the Oracle BI Application for Finance throughout the day. Hence, end users have access to up-to-date information on the Oracle Business Intelligence dashboards as changes occur.

To read more about how real-time data integration can free your critical systems from batch window constraints and how SunGard leveraged Oracle GoldenGate in their data warehousing implementation, I invite you to check out the SunGard case study and the article “Freedom from Batch Windows Using Real-Time Data Integration we wrote for TDWI's What Works in Emerging Technologies publication. As always, you can find more resources on Oracle GoldenGate on our recently redesigned website.

Tuesday May 24, 2011

What’s New in the Data Integration Market?

Data integration is essential to many strategic IT initiatives including master data management (MDM), modernization, and service-oriented architecture (SOA). That’s why it is not surprising that the data integration market continues to grow in its size, importance and impact on enterprise information systems. The technology has evolved from simple extract, transform, and load (ETL) procedures to include real time data movement, bi-directional replication/synchronization along with data quality and data profiling and data services.

 The user base for data integration tools has also evolved from low-level technology for developers and DBAs to high-level environments for operations staff, data stewards, business analysts and enterprise architects. In short, what was once merely the background “plumbing” underlying our information systems now influences just about every aspect of IT, and touches just about every type of stakeholder.

 In a new white paper we have examined the state of the data integration market including the trends in business intelligence, data warehousing, data quality, consolidation, cloud computing and IT modernization initiatives that are driving data-intensive projects. You can access this brand new white paper along with other data integration focused white papers and resources here.

Friday May 20, 2011

Real-Time Data Integration for Operational Data Warehousing

Gone are the days when data warehouses were just for reporting, strategic analytics, and forecasting. Today, more companies are using their data warehouses for operational decision making – and thus more critical to the business. To be able to influence operational decisions the analytical environment needs to be able to stay current with the business events happening right now. Therefore an important requirement is to enable lowest possible latency in which new data is delivered to the data warehouse, ideally in real time.

For example, when a snowstorm hits a certain area, the operational data warehouse can help monitor snow shovel sales and then provide information to help determine whether other stores in the affected areas should move more merchandise to the shelves and also offer other related products at a discounted price. There is a lot less value in reacting to data for a snowstorm that occurred 24 hours ago – or even 6 hours ago.

There are many data integration technologies that serve the data acquisition needs of a data warehouse, however only a few offer real-time data delivery with no impact on source systems’ performance. The challenge for the IT group is to determine what solution or combination of data integration solutions will meet their data delivery and performance needs to help propel the move to operational data warehousing. Such data integration evaluations are aided with the understanding of:

  • Selection Criteria – This should include considerations for acceptable latency, data quantity/volumes, data integrity, transformation requirements, and processing overhead/impact on availability.
  • “Right Time” vs. "Real-Time" – When evaluating solutions, the technology should deliver real-time data capabilities and let the user choose the “right time” as a business decision. Right-time should be a component of decision latency – a user preference, not a technical constraint.
  • Transformations – As the data warehouse approaches real time, transformations ideally should take place within the data warehouse in order to reduce data and analysis latency. This eliminates the need for additional steps for aggregating changed data on a middle-tier server until it is batch processed-- not to mention the TCO savings of not acquiring or maintaining a middle-tier transformation server.

Oracle offers a complete and certified data integration solution for implementing operational data warehouse on Oracle Exadata. Oracle GoldenGate provides low-impact, real-time change data capture and delivery, while Oracle Data Integrator EE provides high performance transformations within Oracle Exadata. Oracle Data Integrator also offers integrated solution for data profiling and data quality to enable analysis with trusted data. Here is a great customer example how Oracle’s products enable the move to operational data warehousing. You can read more about Oracle's data integration solution for operational data warehousing in our white paper. If you would like to read more about how to use Oracle Data Integration products for Oracle Exadata please check out our recent data sheet.


Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality


« November 2015