This is a topic I wrote in six month ago. In the Part I of this series of articles, I mentioned that a warehouse like architecture is required in a heterogeneous environment. I want to elaborate more about this. In the future posts, I will also describe the integration technology I learned for supporting the embedded BI.

Read the rest of this entry >>

To develop or deploy a BI solution for your organizations, you need to have the right people involved in the time time. Here are typical roles involved in a BI data warehouse project.

• Project Sponsor
• Project Manager
• Functional Analyst
• SME
• BI Architect
• ETL Developers
• DBA

The job description and responsibilities are listed in this table: Read the rest of this entry >>

Here is the typical lifecycle for data warehouse deployment project:

0. Project Scoping and Planning

Project Triangle - Scope, Time and Resource.

• Determine the scope of the project - what you would like to accomplish? This can be defined by questions to be answered. The number of logical star and number of the OLTP sources
• Time - What is the target date for the system to be available to the users
• Resource - What is our budget? What is the role and profile requirement of the resources needed to make this happen.

1. Requirement

• What are the business questions? How does the answers of these questions can change the business decision or trigger actions.
• What are the role of the users? How often do they use the system? Do they do any interactive reporting or just view the defined reports in guided navigation?
• How do you measure? What are the metrics?

2. Front-End Design

• The front end design needs for both interactive analysis and the designed analytics workflow.
• How does the user interact with the system?
• What are their analysis process?

3. Warehouse Schema Design

• Dimensional modeling - define the dimensions and fact and define the grain of each star schema.
• Define the physical schema - depending on the technology decision. If you use the relational tecknology, design the database tables

4. OLTP to data warehouse mapping

• Logical mapping - table to table and column to column mapping. Also define the transformation rules
• You may need to perform OLTP data profiling. How often the data changes? What are the data distribution?
• ETL Design -include data staging and the detail ETL process flow.

5. Implementation

• Create the warehouse and ETL staging schema
• Develop the ETL programs
• Create the logical to physical mapping in the repository
• Build the end user dashboard and reports

6. Deployment

• Install the Analytics reporting and the ETL tools.
• Specific Setup and Configuration for OLTP, ETL, and data warehouse.
• Sizing of the system and database
• Performance Tuning and Optimization

7. Management and Maintenance of the system

• Ongoing support of the end-users, including security, training, and enhancing the system.
• You need to monitor the growth of the data.

I talked with people in my team and did some web research. I learned some history about what is now called Oracle BI Applications. The software is actually an accumulated results of more then ten year investments. The "customers" are much more and much bigger then what we know today.

Read the rest of this entry >>

Why is the customer dimension so important?

• A well-maintained, well-deployed conforming customer dimension is the cornerstone of sound customer centric analysis

What are special characteristics of the customer dimension?

• Customer dimension is extremely deep. It may have million rows.
• Customer dimension is extremely wide. It has hundreds of attributes.
• Customer dimension changes often. It is not really a slowly changing dimension. It is sometimes called as rapidly changing monster dimension.
• Keeping the historical images of the customer data is important.
• The data in the customer dimension come from multiple sources. Matching, de-dup, standardization among different sources is critical for successfully conforming the dimension.
• The customer dimension with data may be enriched from the external data.
• Customer data is the source for other dimensions - geography, industry, lines of business, etc.
• You have to comply with the privacy regulation.
• The data from external source may be only licensed for a period of time. The external data need to be associated with your own internal data for analysis for special purposes and the data need to be removed from your database.
• The customer dimension itself is actually a source of fact tables. For example, # of customer living in California with the breakdown by age groups.

The Intelligence Enterprise's weblog has a new article about Oracle 11g.  One thing worth highlighted is the feature he feel impressed about on the advancement of OLAP.

"Materialized views are a technique for speeding multidimensional queries, such as those exploring sales across regions, products and customers. However, as the number of materialized views mounts along with query volumes and complexity, managing those views becomes difficult. In 11g, Oracle is using an OLAP cube to store up to millions of materialized views so they can be managed more efficiently."

One of the problems of the old materialized view is that  you have to know the level you are going to reported by and create the materialized view accordingly.  The benefit of using an OLAP engine is that the aggregation can be available at many different levels andyou do not need to created multiple specific materialized views for storing the summaries.   However, the OLAP cubes was stored in a different technology and populated in different languages.    You need to write codes to map the data from your source system or warehouse to the cubes.  The process needs to be run periodically in order to have the refreshed data from OLAP.  Also, querying the data from OLAP requires a different programming interface, unless you create the OLAP-to-Relational mapping view.

The new feature of 11g seems combining the best from both world and solve all the problems!