5 Ways to Cut Costs for Data Warehousing
By Dain C. Hansen on Mar 09, 2009
Are you trying to better understand what doing "more with less" means for data warehousing? Or are you making the mistake of trying to catch falling knives?
Well if it is the former, hopefully you didn't miss the I-Seminar today, and if it is the latter, well hopefully you have a first-aid kit handy. But in either case the I-Seminar would have done you some good.
But if you missed it, definitely pre-register for April's companion event: 5 Ways to Cutting Costs for Data Warehousing. I'll also be posting the recording of today's webinar once it is available.
Not giving away all 5 ways, but I'll touch on one: Improving Operational Efficiencies through Real-time Data Warehousing.
One important use case that we discussed was how to ensure that data is up to date. In a typical Data Warehouse example, real-time feeds are critical to ensure that the business reports are not looking at stale-week-old data. The more real-time data, the better and more informed your decision making will be.In many such scenarios, change data capture (CDC) plays a key role in keeping data consistently updated without impacting the target or source performance. In addition, these systems draw from a wide range of internal sales, customer, and financial data applications as well as third-party systems. This requires a broad range of data integration connectivity options to support moving data across such a wide variety of enterprise applications.
There are many ways to extract data from a DBMS, including queries, replication, table dumps, storage snapshots, and calls to the API of an application that sits over the database. Change data capture (CDC) is an alternate data extraction method that has recently become of interest, primarily because it enables data integration to operate closer to real time.
CDC can be applied to most database brands, including relational, legacy, mainframe, and file-based DBMSs. A few vendors have built CDC into their products, but many organizations use the data modeling and log capabilities of a DBMS to build their own solutions. CDC has been around for many years, but its ability to solve some of the most difficult data integration challenges is driving interest among IT professionals today.
A simple example of CDC in action follows. Two separate datasources for a web storefront (one for customer data, one for order data) are consolidated into a single data warehouse. To simply update the order details in real-time, only the delta (or set of orders and new customer info) needs to be propagated across to the data warehouse. This does not require moving all the data for both systems. Without CDC, business managers would not be able to see daily trends. In addition, business managers would be forced to wait for the next batch of data to load into the data warehouse before they could look at the results. By then it might be too late to make important informed decisions.
So what does this all have to do with cutting costs? In the I-seminar we reviewed a case study where a large European bank was able to improve their operational efficiencies. They started to see 50% improvements in loading and most importantly maintaining the timeliness of their data. This in turn improved the performance of their business intelligence applications and consequently improved their customer responsiveness.
I've provided you with just one way to reduce costs and improve efficiencies. There are more out there waiting to be discovered. Let us know ways that you're doing more with less and improving on your Data-centric archtiectures. And stay away from falling knives.