Wednesday Apr 29, 2015

Did you know that TABLE_REORG has replace REBUILD_SCHEMA mad REBUILD_TABLES?

Demantra TABLE_REORG procedure. 

can addressed partitioned tables and is more efficient!  Get the latest release at My Oracle Support using bug 17640575.

TABLE_REORG has really replaced and improved the functionality of REBUILD_SCHEMA and REBUILD_TABLES.
It rebuilds the table which is done in primary key order and it moves empty columns to the end of the row.

REBUILD_SCHEMA uses ALTER TABLE MOVE tablespace to reduce chained rows of all tables in the schema.
However, it does not support partitioned tables.

REBUILD_TABLES  is the similar.  It was originally designed for MDP_MATRIX / SALES_DATA, but it can run for all tables and
also for a specific table.  From 2010 it does support partitioned tables.

The procedure MOVE_TABLE was fixed to handle partitioned tables.  It is also out of date, I see ANALYZE TABLE has used parts of the
code (eg for SALES_DATA and MDP_MATRIX).  For an "all tables run" is uses ANALYZE_SCHEMA that does use dbms_stats.GATHER_TABLE_STATS

All tables  - Where the stats value chain_cnt > 0, it does not automatically include SALES_DATA unless 'sys_params','Rebuild_Sales_Table' = 1.

REBUILD_TABLES ( table namel,  stats check,  sales,  all tables flag)

exec REBUILD_TABLES ( null, 1, null, 1) ;    -- With ANALYZE_SCHEMA(100000)  = for none or really old stats
exec REBUILD_TABLES ( null, 0, null, 1) ;

exec REBUILD_TABLES ( null, 0, 1, 1) ;   -- Will include SALES_DATA

For more information see: Troubleshooting TABLE_REORG Package issues - RDF Snapshot drop when process fails + TABLE_REORG Guide MOS Note 1964291.1

Gathering statistics on partitioned tables.  Best practice:

For all 11gr2 environments with large partitioned or subpartitioned objects turn on incremental statistics using this command:

exec dbms_stats.set_table_prefs('OWNER','TABLE','INCREMENTAL','TRUE');

Once this is set for any given table, gather statistics on that table using the normal tool (fnd_stats in ebs or dbms_stats elsewhere).
This first gather after turning it on will take longer than previous analyzes.  Then going forward we will see the following:

1.  The normal dbms_stats or fnd_stats, will only gather statistics on lower level partitions if the statistics on that partition are stale.  This is a significant change.  That is going forward using the default options of a gather command may in fact perform no re-analyze on the actual data if the modifications to the table do not warrant it.

2.  If a subpartition is stale the normal stats will ONLY gather statistics on that subpartition.  The partition for that subpartition will be re-derived as will the global
    statistics, no other statistics will be gathered.

Making this change promises to reduce gather stats by hours in some cases.

For more information: Demantra Gathering Statistics on Partitioned Objects Oracle RDBMS 11gR2, MOS Note 1601596.1

Wednesday Apr 01, 2015

New SQL to Determine Out of Order Ratio and Cluster Factor

Hello!  Please run the following for both mdp_matrix and sales_data.     Questions?  Email at

1. The first SQL will need to be adjusted to accommodate your PK for sales_data and your PK for mdp_matrix.  Adjust the select and from clause to match your PK.   In the case of this sample, the PK was:


-- ShowOOR.sql

     IS_T_EP_SPF ,
     DATA_ROW ,
       IS_T_EP_SPF ,
       ROW_NUMBER ,
         IS_T_EP_SPF ,
       ) C
     ) B
   ) A
 )                               /
 )),3)*100) AS "Out Of Order Ratio %"

-- ShowCF.sql
undefine table_name
SELECT ui.index_name, us.blocks as "Table Blocks", ui.clustering_factor as "Index clustering Factor", ut.num_rows as "Table Rows"
FROM   user_indexes     ui,
       user_tables      ut,
       user_constraints uc,
       user_segments us
WHERE  ui.table_name = ut.table_name
AND    ut.table_name = uc.table_name
AND    ui.index_name = uc.index_name
AND    ut.table_name = us.segment_name
AND    us.segment_type = 'TABLE'
AND    uc.constraint_type = 'P'
AND    ut.table_name      = '&table_name';

Friday Feb 13, 2015

Demantra Table Reorganization, Fragmentation, Null Columns, Primary Key, Editioning, Cluster Factor, PCT Fee, Freelist, Initrans, Automatic Segment Management (ASM), Blocksize....

Hello All!   Why does Demantra require table reordering and null column movement on a, what seems to be, frequent basis?  Your other apps do not require this type of action.  Can this be explained?   I can give you the following.  Let me know if you have questions.  Thank You!   Jeff


Addressing the reason why Demantra requires frequent reordering/reorganization and null column placement, I can explain the reason as follows.  As data is inserted into the Demantra table it will not be in any particular order.  The null columns will be scattered throughout the row.  The main quest is to increase throughput and reduce contention.  Since Demantra is both OLTP and data warehouse the apps DBA must tune accordingly.  Of course EBS and VCP applications have elements of both OLTP and data warehouse but are largely transaction processing at their core.  Applications such as EBS and VCP can typically provide satisfactory response time using typical DBA tuning methods.  Specifically, indexing and proper SQL tuning is plenty to improve and maintain acceptable performance.  However, large tables in data warehouse implementations require unique DBA tuning techniques to reduce IO when accessing the big database tables.  Reducing wait times by eliminating contention is the main goal of table reorganization and the movement of null columns to the end of the row.  The data warehouse requirements of Demantra are no different than other data warehouse applications.

By themselves these are not unique and are implemented in data warehouse applications every day.  For each and every read operation we want to fetch as many relevant rows as possible.  To that end we reorder the large tables according to the PK of each table.  The PK is the main access method to the data according the business requirement.  In the case of Demantra, the PK chosen covers approximately 80% of the data access needed for the main forecasting worksheets.  The rest are addressed using typical DBA tuning methods.  There are new booked, shipped orders inserted into the Demantra world daily, weekly, etc.  When this occurs, the rows naturally loose table/index locality and thus the selectiveness is reduced.  This increases the cluster factor because the number of read operations increases to fetch the desired rows.  The CBO recognizes this and calculates the access method accordingly. 

We want to assist the CBO decision making process as much as possible.  If data was static this effort would be greatly reduced.  Because the Demantra data is dynamic, we need to push the CBO our way to deliver the access plan we desire.  If the table is in Primary Key (PK) order, making your Out of Order Ratio (OOR) low, rows are brought into the SGA in sequential order.  Reducing muli-block reads to process data.  The Cluster Factor (CF) is closely related to OOR.

To review the cluster factor concept, see Demantra Large Database Object Diagnosis and Maintenance Best Practices for Best Performance (Doc ID 1629078.1)

The selectivity of a column value, the db_block_size, the avg_row_len and the cardinality all work together in helping the optimizer to decide whether to use an index versus a full table scan.  If a data column has high selectivity and a low CF, then an index scan is usually the fastest execution method.  Once your large tables are in PK order with a low CF, performance can increase dramatically.  The main goal of table reorganization is to reduce IO when accessing the big database tables.  What it does is two main things:

1. Reorders the table data according to the primary key index.
2. When using the ā€œCā€ which stands for column reordering, it also pushes columns that have no data, nulls, to the end of the table row. This is good thing for tables having a high number of rows as Oracle works in a 254 columns chunks.

Some of the larger Demantra tables have upwards of 200+ columns.  Since every customer uses the Demantra tool in a different manner a portion of these columns will naturally be left as null.  Pushing the null columns to the end of the row decreases data access waits or contention by streamlining the fetch action.  The end result is a column ordered, less fragmented table that will reduce the IO operations for range operations, for example scan data between dates.  The cluster factor will be lower and thus the CBO will look at our desired access method favorably.

Demantra is sensitive to database scans therefore reviewing and implementing sound database performance techniques is essential.  While the EBS and Value Chain Planning (VCP) RDBMS can provide adequate response with proper statistics, indexes, tuning and patching, Demantra brings processing methods to the table that require tighter controls.  I will highlight some of these areas that require tighter control.

1) Tables that have 300+ columns many of the columns are null.  The columns are in the table to represent the family of Demantra solutions.  For
   example Demand Planning, Advanced Demand Planning, Promotion Planning, S&OP, etc.  When the null columns are pushed to the end of the row, the
   read operation becomes streamlined thus increasing performance.

2) Demantra moves significant amounts of data to and from the disk.  It behooves the implementer/Applications DBA to implement a strategy that reduces wait constraints.  Here is a sample of techniques that require strategic implementation attention:

   - Block size 16k or larger
   - PCTFREE set at 30% to 40%
   - FREELIST increased according to your particular implementation
   - INITRANS increased according to your particular implementation
   - Statistics maintained at a 30% sample or larger
   - Parallel Processing at every possible contention point
   - Redo size appropriate to keep switches at 3-4 per hour
   - No Automatic Segment Management (ASM)
   - No Editioning
   - Partitions implemented
   - Large SGA to accommodate multi-block reads
   - Synchronous IO implemented
   - Careful setting of Demantra CBO parameters

In conclusion, it is true that Demantra requires RDBMS implementation strategies that are unique but not uncommon for OLTP and Warehouse systems.
However, VCP requires a unique trick that is unique to VCP.  Creating statistics on empty MLOG$ objects then locking the statistics to the object.
It has been proven that this increases performance by reducing a bottleneck.

Reference Notes:
- Demantra Out of Order Ratio, Table_reorg Procedure, Partitions and Clustering Factor - Manage Your Demantra Database Objects (Doc ID 1548760.1)
- Demantra DB Health and Performance: Oracle Demantra Database Best Practices - A White Paper / Demantra Performance Clustering Factor Out of Order Ratio TABLE_REORG CHECK_REORG (Doc ID 1499638.1)
- Demantra Performance Clustering Factor Out of Order Ratio TABLE_REORG CHECK_REORG (Doc ID 1594372.1)
- Demantra How to Use TABLE_REORG to Reorder MDP_MATRIX in Primary Key (PK) Order Action Plan Version and Later. See 1085012.1 Prior to (Doc ID 1528966.1)
- Demantra Large Database Object Diagnosis and Maintenance Best Practices for Best Performance (Doc ID 1629078.1)
- Demantra Large Database Object Diagnosis and Maintenance Best Practices for Best Performance (Doc ID 1629078.1)

Friday Jul 11, 2014

Demantra Index Cluster Factor Major Update Performance Impact

Hello Everyone!   Please check out the following important information regarding the cluster factor and the impact to the CBO.   Please comment if you like.

Regards!   Jeff

 The Cluster Factor (CF) of an index a very important statistic used by the Cost Based Optimizer (CBO) to determin
the most efficient execution plan.  However, there is a major flaw in the current CF calculation.  Simply put, the CF
is calculated by performing a Full Index Scan, looking at the rowid of each index entry.  If the table block being referenced
is different from the previous index entry, the CF is incremented.  However if the table block being referenced is the same
as the previous index entry, the CF is not incremented.  So the CF gives an indication of how well ordered the data in the table is
in relation to the chosen index entries.  

The lower the CF, closer to the number of blocks in the table vs the number of rows, the more efficient it is to use the index as
less table blocks would need to be accessed to retrieve the necessary data via the chosen index.

So far so good.  However, what if part of the incoming row is stored in block one and the remainder in block 2?  When the row is
required, block 1 and block 2 will be read into cache resulting in near zero waiting for the entire row.  OK, fast forward, what if
the table has 100m rows and imagine that this situation occurs 30% of the time.  Your current CF method is most likely skewed
and incorrect.  

Now for the good news, bug 13262857 - INDEX CLUSTERING FACTOR COMPUTATION IS PESSIMISTIC.  The computation of the index clustering
factor in dbms_stats package is pessimistic about the caching ratio of the table blocks.  It assumes that at most one block from the
table is cached.

This is an enhancement to allow a user to specify the number of blocks that dbms_stats package will consider when gathering the index clustering
factor statistics. Prior to this enhancement dbms_stats assumed 1 and it still does after the enhancement.  This enhancement allows the user to
specify a value between 1 and 255.  There is also an AUTO option which if specified then dbms_stats will use 1% of the table blocks up to
0.1% of the buffer cache size, in blocks.  

The new CF approach is currently available with patches that can be applied on both Exadata databases and Oracle versions, and
The Patch ID is 15830250.

Once applied, there is a new statistics collection preference that can be defined, called TABLE_CACHED_BLOCKS.  This basically sets the number of
table blocks we can assume would already be cached when performing an index scan and can be ignored when incrementing the CF during statistics gathering.
The default is 1, as stated, the same as today.  But can be set up to be a value between 1 and 255, meaning that during the collection of index statistics
the process will not increment the CF if the table block being referenced by the current index entry has already been referenced by any of the prior 255 index
entries, if set to 255.  It basically sets the appropriate parameter in the sys_op_countchg function used to calculate the CF value during statistic gathering
to not increment the CF if the current table block has already been accessed ā€œxā€ index entries previously.

Apply the patch, change the TABLE_CACHED_BLOCKS preference for this table and re-calculate the index statistics:

SQL> exec dbms_stats.set_table_prefs(ownname=>dm, tabname=>'SALES_DATA',

pname=>'TABLE_CACHED_BLOCKS', pvalue=>50);

PL/SQL procedure successfully completed.

SQL> EXEC dbms_stats.gather_index_stats(ownname=>dm, indname=>'SALE_DATA', estimate_percent=> null);

PL/SQL procedure successfully completed.

SQL> SELECT t.table_name, i.index_name, t.blocks, t.num_rows, i.clustering_factor
2  FROM user_tables t, user_indexes i
3  WHERE t.table_name = i.table_name AND i.index_name='SALES_DATA';

Although I found this bug internally, many thanks to several on the WWW for their comments.


This blog delivers the latest information regarding performance and install/upgrade. Comments welcome


« August 2016