It is still happening... some IT departments prefer to handcode data quality and ETL processes. Programming is labor of staff, and those costs are made anyway. Buying a tool actually costs money and that is hard to justify. And some organizations still suffer from the NIH syndrome (Not Invented Here) and think they can do better themselves. "Standard tools are for sissies, here we build our own stuff that is faster than anything else!" I recently witnessed a few of these cases, and I just can't believe this to be true in 2007. First of all, much of ETL, DQ and other functionality simply sits in the database these days, and it is not needed to spend hundreds of thousands of dollars on specific technology. But that's not even the point. It is about understanding IT strategy.
ETL, data quality, master data management and other related areas are infrastructure. They should not be seen as a tool to support a single project, such as a BI or CRM system, but they should support the complete flow of information within and between systems and business processes. Billing systems getting data from transactional systems require ETL and the likes, system conversions do, Financial Consolidation does, etc etc etc. The return on investment on configuring a tool is high and is achieved in a very short time. The first project may be harder, but once configured, results are booked faster. I think the break-even point usually is reached at the 3rd project or application already.
A good IT strategy distinguishes the areas where IT is a standardization game, usually on the infrastructure level. Where that is the case, you simply take the tool that comes with your database or middleware provider. Then there are specific areas where there can be competitive difference, probably at the analytical level. There it pays off to be smarter than others and build a more heterogeneous environment (that in Oracle terms should be hot-pluggable). Developing stuff yourself can still be a cool thing, as long as you can really make a difference, making your organization smarter and more agile. Focus your skills on that, and use standard tools for everything else, as the necessary intelligence is built in there already.
frank
