Thursday Mar 13, 2014

JavaScript Move Validation

Today, let's talk about move validations. They are a kind of real-time validation you can create. Before the introduction of Dynamic Scripting, it wasn't really possible to make use of move validations (because they only fired after the move) so they often get overlooked.

With Dynamic Scripting, a move validation actually gets fired twice in a two-phase execution. The "move.IsPre" and "move.IsPost" tell you which phase you are in, so a simple if/else check allows you to separate your script.

1. IsPre - Before the move takes place

This phase gives you the node as it exists under the old parent and is your opportunity to collect information to be checked in the second phase. Record any values in your "move.Values" object, but be aware that any complex objects like nodes or versions will get "flattened" to simple strings because the objects they represent change state after the move, so be sure to dig out any actual property values you are interested in.

2. IsPost - After the move takes place

This is what we normally think of as the regular validation phase and happens after the move. The node is now under the new parent, which means any inherited property values have changed, calculated values are updated, etc. That means you can compare the new values to any old values you saved in the "move.Values" object.

In both phases, you have access to "move.OldParent" and "move.NewParent" so you can check any property values or structure you like of the old parent and new parent both before and after the move. For example, you could stop the move if the new parent already has too many children in the pre move phase; or you could stop the move if the value of a derived property on the old parent changed after the move.

I hope with these new capabilities customers will find that Move validations are a powerful tool for data quality.

P.S. Merge validations are similarly enhanced with JavaScript, though there is only the one traditional phase of execution. You have access to the merge target (merge.Target) and a flag to indicate whether the merge is an Inactivate or a Delete (merge.IsInactivate/merge.IsDelete).

Monday Feb 17, 2014

Database Recommendations

If you are deploying a reasonably large environment for Data Relationship Management or Data Relationship Governance, I recommend following industry best practices for your database server:


  • Provision as much RAM as possible
  • Locate the servers close together with a low-latency high-bandwidth network
  • Place the temp database, index files, and data files on separate storage partitions.
    • If possible you can even split tables onto separate partitions; the transaction history and global/local property value tables are good candidates.
  • Use high-bandwidth storage (such as fiber-channel) as many in parallel as you can
  • Use Solid State disks

If you haven't guessed, the them is to spread the disk load onto as many separate SSDs as you can, then to pair the SSDs into stripes behind high-bandwidth links. You might be surprised at how much this can affect your database performance for all applications, not just DRM.

Monday Jul 01, 2013

SQL Server Database Settings

For those using Data Relationship Management on Oracle DB this does not apply, but for those using Microsoft SQL Server it is highly recommended that you run with Snapshot Isolation Mode. The Data Governance module will not function correctly without this mode enabled. All new Data Relationship Management repositories are created with this mode enabled by default.

This mode makes SQL Server (2005+) behave more like Oracle DB where readers simply see older versions of rows while a write is in progress, instead of readers being blocked by locks while a write takes place. Many common sources of deadlocks are eliminated. For example, if one user starts a 5 minute transaction updating half the rows in a table, without snapshot isolation everyone else reading the table will be blocked waiting. With snapshot isolation, they will see the rows as they were before the write transaction started. Conversely, if the readers had started first, the writer won't be stuck waiting for them to finish reading... the writes can begin immediately without affecting the current transactions.

To make this change, make sure no one is using the target database (eg: put it into single-user mode), then run these commands:

ALTER DATABASE [DB] SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE [DB] SET READ_COMMITTED_SNAPSHOT ON

Please make sure you coordinate with your DBA team to ensure tempdb is appropriately setup to support snapshot isolation mode, as the extra row versions are stored in tempdb until the transactions are committed.

Let me take this opportunity to extremely strongly highly recommend that you use solid state storage for your databases with appropriate iSCSI, FiberChannel, or SAN bandwidth. The performance gains are significant and there is no excuse for not using 100% solid state storage in 2013.

Actually unless you need to store petabytes of archival data, there is no excuse for using hard drives in any systems, whether laptops, desktops, application servers, or database servers. The productivity benefits alone are tremendous, not to mention power consumption, heat, etc.

Thursday Jun 27, 2013

Scripting Part 1

Dynamic Scripting is a large topic, so let me get a couple of things out of the way first. If you aren't familiar with JavaScript, I can suggest CodeAcademy's JavaScript series. There are also many other websites and books that cover JavaScript from every possible angle.

The second thing we need to deal with is JavaScript as a programming language versus a JavaScript environment running in a web browser. Many books, tutorials, and websites completely blur these two together but they are in fact completely separate. What does this really mean in relation to DRM? Since DRM isn't a web browser, there are no document, window, history, screen, or location objects. There are no events like mousedown or click. Trying to call alert('hello!') in DRM will just cause an error. Those concepts are all related to an HTML document (web page) and are part of the Browser Object Model or Document Object Model. DRM has its own object model that exposes DRM-related objects. In practice, feel free to use those sorts of tutorials or practice within your browser; Many of the concepts are directly translatable to writing scripts in DRM. Just don't try to call document.getElementById in your property definition!

I think learning by example tends to work the best, so let's try getting a list of all the unique property values for a given node and its children.

var uniqueValues = {};

var childEnumerator = node.GetChildEnumerator();
while(childEnumerator.MoveNext()) {
    var propValue = childEnumerator.GetCurrent().PropValue("Custom.testpropstr1");
    print(propValue);
    if(propValue != null && propValue != '' && !uniqueValues[propValue])
        uniqueValues[propValue] = true;
}

var result = '';
for(var value in uniqueValues){
    result += "Found value " + value + ",";
}
return result;

 Now lets break this down piece by piece.

var uniqueValues = {};

This declares a variable and initializes it as a new empty Object. You could also have written var uniqueValues = new Object(); Why use an object here? JavaScript objects can also function as a list of keys and we'll use that later to store each property value as a key on the object.

var childEnumerator = node.GetChildEnumerator();
while(childEnumerator.MoveNext()) {

This gets an enumerator for the node's children. The enumerator allows us to loop through the children one by one. If we wanted to get a filtered list of children, we would instead use ChildrenWith(). When we reach the end of the child list, the enumerator will return false for MoveNext() and that will stop the loop.

    var propValue = childEnumerator.GetCurrent().PropValue("Custom.testpropstr1");
    print(propValue);
    if(propValue != null && propValue != '' && !uniqueValues[propValue])
        uniqueValues[propValue] = true;
}

This gets the node the enumerator is currently pointing at, then calls PropValue() on it to get the value of a property. We then make sure the prop value isn't null or the empty string, then we make sure the value doesn't already exist as a key. Assuming it doesn't we add it as a key with a value (true in this case because it makes checking for an existing value faster when the value exists).

A quick word on the print() function. When viewing the prop grid, running an export, or performing normal DRM operations it does nothing. If you have a lot of print() calls with complicated arguments it can slow your script down slightly, but otherwise has no effect. But when using the script editor, all the output of print() will be shown in the Warnings area. This gives you an extremely useful debugging tool to see what exactly a script is doing.

var result = '';
for(var value in uniqueValues){
    result += "Found value " + value + ",";
}
return result;

Now we build a string by looping through all the keys in uniqueValues and adding that value to our string. The last step is to simply return the result.

Hopefully this small example demonstrates some of the core Dynamic Scripting concepts. Next time, we can try checking for node references in other hierarchies to see if they are using duplicate property values.

Wednesday May 15, 2013

Script Properties and Validations

A new feature for 11.1.2.3 is the ability to use ECMAScript/JavaScript for properties and validations. Here is an example of a validation written in script:

As you can see, scripts can contain comments, declare variables, and do a wide variety of things that are simply not possible with the formula system. Validations can return parameters which will be inserted into the failure message, allowing a validation to return a specific failure message rather than a generic message. Script properties can be declared at the version or hierarchy level, the first time DRM has supported a calculated property at those levels.

You can find more information about the script object model in the admin guide. The format of the guide isn't ideal for API documentation; I hope to get a better format and some examples available soon.

Monday Apr 29, 2013

Release 11.1.2.3

DRM Release 11.1.2.3 is now available and there are several exciting new features to discuss:

  • Import from Database Tables - this has been a long-requested feature that allows you to skip creating a text import file
  • Job Manager - job results are persisted in the background even when you log off, or you can simply close the tab and navigate elsewhere in the UI without losing access to the job.
  • Dynamic Scripting - property derivers and validations can now be written in JavaScript! I will have more to say about this in future blog posts, but you have access to ECMAScript-standard JavaScript features. Dynamic Scripting is also enabled for version and hierarchy properties, allowing derived properties at this level for the first time. Dynamic Scripting also supports Move validations with a pre and post move phase, and the validations can calculate a custom error message rather than just returning the same generic error for every failure (e.g. "Prop X was 6 but must be less than 5" instead of "Prop X, Y, Z, and A must be less than five when parent is K"). This is all in addition to the existing Node properties and basic validation types.
  • Data Governance - you can now create workflow paths and manage approvals, all without leaving DRM. Derived properties are calculated and validations run at each stage, examining values as they will be when committed but without affecting the system until the request is approved. Best of all, access rights to submit, claim, enrich, approve, or commit a request are driven by the security set on your hierarchy itself (for example the business owner of a hierarchy can be the final approver).
There are many other changes, bug fixes, and enhancements. I will be blogging about many of these things over the next few months so stay tuned.

Monday Apr 30, 2012

New features in 11.1.2.2

  • ASCII Code Pages
    • You can now select from a list of ASCII code pages when dealing with importing/export files to support legacy system that can't process Unicode.
  • User Interface Localization
    • The UI is now localized in these languages!
      • French
      • German
      • Japanese
      • Korean
      • Simplified Chinese
    • Property Type Modification for Saved Property Definitions
      • You can change the property type for an existing property definition (eg: Defined to Dervied) to improve customization of properties loaded from application templates and to preserve any relationships to other metadata objects which depend on the property to function properly. The primary consideration when changing the property type is if the property is being changed from one that can store data to one that cannot (defined to derived or lookup). In this situation, the user is warned that data will be lost and must confirm the change.
    • Formula Based Validations and New Formula Functions
      • A new Formula validation class may be used to enforce business rules that require complex logic which can be expressed in a Data Relationship Management formula. This validation class bypasses the need for a derived property definition to be created which may greatly reduce the complexity of an application configuration. New formula functions are also available to perform advanced calculations for derived properties and validations (more on this in the future)
    • Record ID for Exports
      • When exporting to a database table, a record number is auto-generated for each record being output to preserve the order of the records as they are exported.
    • Batch Validations for Export
      • Validations can be run prior to executing a Data Relationship Management data export. If any of the validations fail the export/book will be stopped. You can choose to run the assigned validations or a list of selected validations (which can be saved as part of the export definition).

        In addition, the LRO engine now obtains a lock for the duration of the validations and export process to ensure that the data is consistent, including a book. It also runs the book validations in parallel, which can greatly speed up the validation phase.
    • Hyperlink Enabled Properties
      • A new Hyperlink property data type is available to manage URLs which refer to Web resources. The URLs display as navigable hyperlinks in the Data Relationship Management Web Client. The Memo and FormattedMemo data types can also support hyperlinks embedded in regular text. All of these property data types support the option of defining a label for each URL to improve on-screen display of complicated or lengthy URLs.
    • Navigation to Hierarchy and Node via URL
      • You can directly access a hierarchy or local node in the Data Relationship Management Web Client from an external document or program using a context-senstive URL. The URL is accessible as a system property for a hierarchy or local node for easy reference and copy-paste to another application. If Single Sign On (SSO) or anonymous access is enabled, the user will directly navigate to the version, hierarchy, and node with which the URL is associated. If not enabled, the user is directed to the login page and upon successful log in, is navigated to the hierarchy or node.
    • Copying Nodes and Properties Across Versions
      • You can copy nodes from one version to another. The Put Node and Insert Node features let you select a node from a version other than the current version. On Compare result page you can drag and drop and a node from one version to a different version. For each scenario where a node is being copied across versions, you have the option of copying properties and descendant nodes from the source to the target version. You can also copy node properties across versions from one node to another node.

        There are a few restrictions on this feature - mainly none of the nodes can exist in the target version (shared nodes will not be created).
    • Hierarchy Group Properties
      • Application Administrators can create hierarchy group properties to enable users to group hierarchies in multiple ways based on different contexts. A hierarchy can be assigned to different groups for different properties but can only be associated to a single group per property. The Hierarchy Tree toolbar has a new drop-down list called Group By for selecting a hierarchy group property. Only the hierarchy group properties that the user has access to are available in the Group By drop-down list.
    • Version Variables
      • Data Creator, Data Manager, and Application Administrator role users can utilize variables to refer to versions in a Data Relationship Management application. Version variables may be used to dynamically refer to different versions over time or provide an easy method of selecting a version from a large list of versions. Version variables can be saved with user metadata objects such as queries, compares, imports, blenders, and exports. In all areas of Data Relationship Management, a version variable may be used in the same manner as a version name. The previously available DefaultCurrentVersion and DefaultPreviousVersion system preferences are now accessible as version variables.
    • Version Backup and Restore
      • Data Manager role users can archive or migrate data from one Data Relationship Management application to another using version backup and restore. You can include one or more versions in a binary backup file that is written to an external connection defined by an administrator. The backup file may be later restored into the same or a different application. The Batch Client also supports version backup and restore for automation purposes.

        Please note that this backup does not include the transaction log and cannot be used to produce true As-Ofs. Please do not use this as a substitute for database backups! It is more like taking a snapshot of the version and copying that snapshot to another application (or sending to support to help diagnose a problem).

Thursday Apr 19, 2012

Performance Question

Quick question for those of you using DRM: Are you seeing any performance issues? If so, please drop me a line with some information. Things like:

- What version of DRM are you using?
- What browser are you using?
- Where is your database located (same data center or across a WAN?)
- Is it slow responsiveness in the UI, the length of time for a job, or something else?
- What area is giving you trouble? (Versions, Exports, etc)

Your feedback is greatly appreciated!

Thursday Apr 12, 2012

EPM Release 11.1.2.2 GA

Just a quick note: DRM 11.1.2.2 (TR PS2) is now GA. There are a number of improvements, bug fixes, etc. I hope to write about some of them soon including localization in several languages, domains, validations on export, etc.

Tuesday Feb 28, 2012

Upgrading from 11.1.1 or earlier

Just a quick note: If you are upgrading from 11.1.1, 9.5, etc then you might see large database sizes in the target database due to DB transaction logging. You can avoid this (and speed up the upgrade) by setting smaller redo logs on Oracle DB and turning off ARCHIVELOG. On SQL Server you can set the database to SIMPLE recovery mode. In both cases you only need to run in this mode during the Upgrade process then you can return to regular operation. Normally I wouldn't recommend running this way but there is little risk here because DRM isn't running with live transactions on the target DB and if anything goes wrong you can just restart the upgrade to a new blank target DB.

Wednesday May 11, 2011

Note on Validations Bug

A quick note on Validations: For versions 11.1.2.0 - 11.1.2.1.101 the GlobalPropQuery, RequiredField, VersionUniqueProp, and Version2UniqueProp validation types should not be used. There is a bug that will cause these validations to return "success" when in fact the validation should fail.

We apologize to anyone affected. If this is a critical issue for you (or you would just like to be notified when a fix is available) please open a support ticket.

In the mean time for all but the GlobalPropQuery validation you can bypass the issue by creating a derived property to do the equivalent calculation. Support should be able to assist you in setting this up if you run into any issues.

Thursday Apr 14, 2011

What's in 11.1.2.1.0.101?

[Read More]

Friday Apr 08, 2011

11.1.2.1 Now Available

[Read More]

Tuesday Jan 11, 2011

Url Navigation

[Read More]

Tuesday Jan 04, 2011

What's New In 11.1.2.1 (Talleyrand SP1)

[Read More]
About

Writing about Oracle Hyperion Data Relationship Management

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today