By user702295 on Mar 12, 2014
Hello Demantra Customers! There is a new white paper that discusses large data set loading.
See MOS Note 1634925.1
Recently we have encountered a few scenarios where customers tried to process huge amounts of data within a Demantra process.
- Customer wanted to load over 50M records via EP_LOAD process on weekly bases.
- Customer wanted to process millions of rows via BLE process on daily bases.
- Customer wanted to load via integration interface 40M rows on weekly bases.
In all of the above the customer complained about the system inability to process the amount of data, either being too slow or it simply could not be completed at all meaning the process erred out.
In all of the above examples an escalated defect was logged, and development worked troubleshooting the problem.
An additional common theme between all the above examples and many more is that there was no real need to have a product fix although performance improvement opportunities may have been identified, they were not the final solution that addressed the problem.
What helped was understanding that there is no real need to process all the data and recognizing the data that really needed to be loaded. The data could have been loaded in a more efficient way using some best practices and creative thinking.
The scope of this document is to provide some guidance how to troubleshoot such customer situations.
The document will focus on three main areas:
- Learning and understanding the business needs
- Understanding the actual data that needs to be loaded, understand the gap between this number and the number of rows that the customer actually tries to load.
- Provide some best practices that can help the customer work with the data efficiently.
This document will not deal with the initial Data load processes although some of the best practices can be adopted for such situations as well.
We will illustrate the concepts in this document using three real life examples of service requests and/or defects that were logged on behalf of the customer.