Virtual Identity Dialogue

Since a number of Oracle sales folks and partners read this blog - I wanted to help get the word out.

Oracle maintains a system that allows you to demo live software from a hosted service called TDS. We have just released several new IDM related demos including OVD, OIF (federation), OAM Suite (including OAAM & entitlements via OES) and Oracle Enterprise SSO (aka SSO for client-server).

The benefits of using TDS for demos is that:

1 - All demos contain scripts (helpful if you need to demo a product you are not intimately familiar with, in particular how to give a good story to accompany the demo) but are also live. Thus can "go off script" if you need to.

2 - All demos have been developed with assistance of Product Management - thus meet our criteria of what we want to see in demos. In fact most are the same demos we have shown at conferences like OpenWorld - thus are "proven".

2 - Don't need to keep lugging around vm images - in particular for demos that may require more resources than your laptop has.

Unfortunately TDS is not currently open to non-Oracle staff (except for Oracle partners). But if you are a customer & have been wanting to see a demo of one of these products - please contact your account rep to ask to see one of these demos.

Update - here is the page for partners.

Over the holiday break someone sent me a comment asking to update the Virtual MemberOf plug-in to support the condition where a person's member group was actually a member of another group.

But I also encountered a customer requirement that needed to see list of Dynamic Group memberships.

So I added that too.

The plug-in is here. 

Ash Motiwala has some follow-up questions to my previous responses to him.

After seeing my posted results he asks:

"If so, then why does anyone use a persistent cache? Anyone?"

Ash - this question has perplexed me as well :).

Then he requote's Blink Technologies comment - which I will respond to here:

I thought the whole point of the cache is to lighten the load against the system as a whole. It's a compromise of data freshness for performance. Plus the entire point of a cache is to "cache" frequently used data, of course depending on the algorithm used (LRU, MRU, etc.). I also assume that the cache is adjustable and can have specific timeouts for freshness. I think for a highly trafficked directory this is a great trade-off.

I thought the whole point of the cache is to lighten the load against the system as a whole.

This is not actually the case most of the time. The purpose of a cache is to reduce latency.

 

It's a compromise of data freshness for performance.

Correct. Except remember when we are talking about virtual directories - 1) the sources often have their own caches (to reduce latency) 2) the applications usually cache the lookups they do against a directory (e.g. they lookup once against OVD & that is it for the duration of the user's application session) 3) You are often using a directory in security applications thus you must pay extremely careful attention to that the data-freshness compromise. Remember identity data is not like cheese which gets better with age. It's more like raw milk - unless treated carefully - spoils quickly (with potentially devastating results). 

Plus the entire point of a cache is to "cache" frequently used data, of course depending on the algorithm used (LRU, MRU, etc.). I also assume that the cache is adjustable and can have specific timeouts for freshness.

Yes - caches should be caching frequently used data. But let me go back to my initial point - CACHE ALREADY HAPPENS AT THE DATA-STORE AND APPLICATION LEVEL.

So any further caching at the virtual directory level - doesn't necessarily add any benefits.  But in the rare case where you might need to access a data-store that doesn't do its own caching - OVD provides an in-memory cache plug-in that supports very granular configuration.

Ash gave me a heads up that there were some interesting comments on his blog in regards to caching and asked if I would give my thoughts. 

Todd Clayton gave an example which I can summarize as follows:

• A company has HQ in Chicago and several branch offices which include customer facing applications
• Each local branch has its own LDAP server for authentication for internal & customers
• There is a proprietary database containing customer entitlement information maintained in Chicago
• A virtual directory will be used to span both the LDAP & to convert entitlements into LDAP accessible data

But then makes a giant leap in assuming the data would need to a persistent cache to be viable for a virtual directory.

Here is how I would answer this - if I was given this as a question from a prospective customer.

First I would say is that just because the database is not local to the branch, does not automatically mean the database needs to be copied to the local branch in order for a virtual directory to make effective use.

For sake of argument - let's assume branch 1 manager says the data link to Chicago is unreliable for persistent network connections, so need to assume that the database will not always be available for persistent connections

To which I would say - "Ok, that's fine. One of the benefits of Oracle is that we have a number of solutions to this type of issue. So let's do some more investigation to see what is the best solution."

Because based on my experience one of three possibilities is really on the table. And each has an optimal solution:

Possibility 1 - Company policy states that the database data cannot be stored in any location except HQ. So while it is possible to query the data - it cannot be stored in the branch

Solution 1 - Use OVD in-memory cache so that data is not stored but in case an application makes repeated queries against the database during a customer log-in, those queries won't make the jump across the WAN. The branch understands that because of company policy sometimes that database will not be available. In those situations - customers might be denied access. But branch understands this is only possible solution unless either data-link is improved or company policy changes.

Possibility 2  - The data can be copied but has never been done because HQ politics have prevented a solution. Additionally the branch(es) needs to run their own database reports against the data (besides using LDAP for authorization). And would prefer to solve that problem as well.

Solution 2 - Use Oracle Times-Ten In-Memory database to make the data available via both the virtual directory and to local SQL-based reporting applications. Benefit of Times-Ten is that its blazingly fast (it's optimized to store data in-memory as primary data-store) and because it's designed to be non-persistent - don't have to worry about local data-storage (in particular if the HQ database is an Oracle DB).

Possibility 3 - The data will only be used for LDAP authorizations. The data-link is truly unreliable and thus the data really need to be stored locally to make this operation viable

Solution 3 - Oracle Virtual Directory is just one component of Oracle Directory Services (which is how its licensed). Oracle Directory Services includes Oracle Internet Directory & Directory Integration Platform. Because this data is mission critical to the branch operations and because it will only be used for LDAP operations and because the data-link is too unreliable for direct connections - DIP can be used to store the data (& keep it updated locally) into OID.

Possibility 4 - The network manager enters the meeting and announces that new private high-speed fiber line is going live next Monday. Thus data connectivity problems will be resolved.

Solution 4 - After go-live discover the SQL queries even with join are resolving faster than the application takes to display their UI. Thus no need to actually cache any data.

In summary:

• Just because a data-store is remote does not automatically mean "must be copied locally"
• Even if data needs to be copied locally - need to pick a data-store/synchronization solution that meets all of the needs of the customer requirements

Ash asked me to comment on a couple of responses to his blog.  This post is in response to the post made by the Identity Data Delivery blog.

First off, while it’s true there are times when cache makes no sense, there are other times when it really does.  Cache is used everywhere in your PC, software, servers, EVERYWHERE —so arguing against cache seems completely strange to me.

Nobody is arguing against cache - as I have mentioned - OVD leverages the benefit of EXISTING caches on the data-store and applications. Thus any additional cache benefits are not going to have a large pay-off in the most common non-telco IT case. And this is just talking about a traditional, in-memory cache as opposed to actually copying, synchronizing, securing, getting enough hard-drive space and backups that a persistent cache requires.

Second, I always find it funny to hear the arguments against having more options. Why argue against choice and options? I can cite many projects that have been deployed with caching using virtual directories, and yes, this includes "persistent" cache.

Nobody is arguing on options. The specific question is "do you need a persistent cache in the most common virtual directory scenarios?". And the answer is a definite no. And as I pointed out here - even if you think you might need a type of persistent cache - it's better to make that an enterprise architectural choice so that you pick the right solution.

Third, cache is necessary because when merge multiple tables (join) across different databases (or directories for that matter), the results are just not fast enough for any type of security application. Anyone familiar with databases should understand this quickly. Once you join several objects or tables, the response rate of the source is dramatically reduced. The joins necessary to create views are sometimes too complex to do on the fly for most directory-enabled applications, such as would be common for IdM/security. This, in my mind, is a key functionality of virtual directories after aggregating sources for a common protocol.

Yes - aggregating heterogeneous sources without copying them into a central store is the primary reason to use a virtual directory. And we have many production instances where data is spread across multiple sources and have never needed a persistent cache to meet performance requirements. This is because modern data-sources are very fast, applications are better at keeping session data and OVD can optimize the queries so that if a data-store is not going to contain the answer, it's not searched.

Fourth, 2 to 5 milliseconds can be a big deal, and cache is essential to eliminating that lag. Think about it, if I have to search for a member in a directory, and then search a database table for additional attributes to join to this object — do you really think it will perform at close to the same speed? And that is with just two sources...imagine the performance hit you’d take by adding additional sources and multiple join operators. 

I do not know of any production scenario where the overhead of 2-5 milliseconds is an issue. OVD is already being used in production telco scenarios and we're actively involved with several Unified User Profile type applications for the cable/ISP/telco market. And in every case 2-5 milliseconds is not only OK - it's a magnitude lower than the specified requirement (usually around 20-25 ms). Also in scenarios where joins are involved - the data can usually be pre-fetched (for example notification of incoming customer call can prompt a notification to OVD to fetch the customer record prior to any application using the data. Thus it's cached in memory but not stored. Remember the argument here is on "persistent cache" not cache in general).

When your directory is expected to perform at 8000 queries a second, adding 2 to 5 milliseconds can be a VERY big deal. OK, let's keep the math simple and take a closer look what the problem is…

• I have a directory that performs at 5000 q/sec
• That translates into .2 milliseconds per query (5000/1)

• Your "overhead" is 2 milliseconds (the best performance cited)
• My queries now take 2.2 milliseconds (11 times slower)
• Now instead of 5000 q/sec when I access my directory I get only 455 q/sec

The blogger made a fanciful leap - that 2 ms time I mentioned is consistent even around several thousand queries per second. We have a customer in EMEA who authenticates all of their 3G phones via OVD without any cache in OVD. So this is a false response.

Fifth, the idea that you compromise the "freshness" of data for the sake of speed misses the point about what sort of information we’re dealing with here. We’re dealing mostly with identity using directories (people and other objects), and the identities themselves do not change very often in comparison to other data, such as transactions where updates and write operations are more common than search/query. For example, in your bank account, your “identity” information (name, address, phone, pin number, passwords, etc) changes far less your balance and activity.

If the only value a virtual directory provided was real-time access to data - then this would be a valid point. However, the primary value a virtual directory brings is that the data is not data-freshness. Rather it's that you get an aggregated view of identity data without needing to consolidate it into a single data-store. If your virtual directory requires a persistent cache then it is not living up to the promise of a virtual directory and instead is selling you a meta-directory disguised as a virtual directory.

And if you want a virtual directory or similar type of centralized identity store - that is fine. It's a valid architecture that is not even necessarily an either/or problem (e.g. either use meta or use virtual).

And at Oracle we have products that can help you solve that type of problem - it's just that they are not virtual directories.

James McGovern has asked me to respond a couple of times to his follow-up to an earlier thread.

Let's look at one of mark's quotes:" Let's take a step back here. Sun did not open-source LDAP :). They have an open-source project that wrote from scratch an open-source ,storage-based LDAP server in Java." which says that Sun appreciates that open source is not just about dumping dead products on the market, but understands that participating in a larger community has immense value.ust because a product is open-source does not mean there is community or participation in that community. Or that having a community has "immense value".

First let's clarify something "open-source DOES NOT EQUAL community". It only means you can see the source of the software you are running. I mean I can see the engine components to my Chevrolet Cavalier but there is not any community around Cavaliers.

And just because a product is not open-source doesn't mean there isn't community. Oracle has a large community around all of our segments - some of which is open-source though most is not. 

I would state that no other enterprise-software company has done more to build & leverage its community more. This ranges from Oracle Mix and Oracle Wiki all they way to the new Oracle "My Support" system which integrates Metalink, forums and related features to help get problems solved more quickly. And of course the old standards OTN Forums and Oracle blogs.

Thus while you may not always be able to see our code - I would disagree with the assessment that just because we don't have everything open-source we're not involved in community. 

Sun showed leadership in allowing the community to make the choice of how the product will evolve without the overhead of sales folks filtering out ideas before they reach product managers.

A good product manager spends a great deal of time listening to their customer-base. This includes talking directly to customers in pre-sales, during implementation as well as reviewing support items. Sales people do play a valuable role here because they can help provide a way to speak to trends.

Additionally this is why internally we encourage all of our PMs as much as possible to participate in things like blogs, forums and new things like Oracle Mix to get feedback.

But trust me -  I don't think most PMs suffer from lack of feedback from customers.

The biggest detriment is that we're not always able to publicly discuss roadmap and timelines. This is because as a public company there are very strict revenue recognition rules. Even open-source can't save you from that boogeyman.

And finally I would also point out sometimes to innovate - you have to NOT listen. As Henry Ford once said when asked if any customers had told him to make cars - his reply "If I had asked anyone they would have told me to make faster horses".

Now, let's compare this line of thinking to Mark's comment: At the moment we are still able to grow the adoption of OVD (and OID), are able to improve upon the core product via customer feedback and have a plug-in API that allows for customers (whether themselves, partners or Oracle consulting) to extend the product to meet their needs - so I don't sense a valid reason to open-source OVD.. Does anyone see a difference in openness?

If the question is only "can I see the source" then this argument is valid. However, if the question is one of "can I get my problems solved quickly and have a way to communicate with development to help guide direction of the product" then I would suggest there is very little substantial difference.

Here is another quote from Mark: Microsoft has produced open specifications, a few examples and started the Information Card Foundation (which we are a member of) to help drive adoption of Information Cards. I would argue we are on the same path on IGF via Open Liberty. Of course, he conveniently misses talking about the fact that Microsoft also funded implementations of information cards for platforms such as PHP and Java, languages obviously non-Microsoft. So, can we expect Oracle to fund IGF libraries for non-Oracle languages such as Smalltalk, Ruby on Rails and .NET?

I didn't miss anything.

First - IGF (unlike Information Cards) is an open-standard driven by Project Liberty. While Phil Hunt is the primary contributor and an Oracle employee - it has been developed within the framework of Liberty. Since IGF was largely influenced by both OVD and Project Higgens - both of which are Java based - the initial implementation were done in Java. However, when I wrote the initial spec for what has becomes ArisID Beans - I consciously wrote the API so that it would be as language neutral as possible (I was heavily influenced by the only counter example out there which is XML DOM API).

Second - Unlike Information Cards there is not as much of a pressing need for non-Java API at the moment. In particular since most of the world is moving on to things like REST and SOAP - IGF is going to have to adapt to those constructs. And until that is worked-out, doesn't make a heck of a lot of sense to do other implementations. Not to mention if there is a REST/SOAP interface - the need for "native" implementations is greatly reduced.

Third -  Microsoft had a stronger need to demonstrate its openness on this after the disaster that was Passport/Hailstorm. I think IGF has the luxury of being able to take more time.

(also - really Smalltalk?  Why not COBOL? or Logo? :))

Mark also previously blogged on How Oracle can help you write more secure code. I wonder if he is familar with the Open Web Application Security Project? Notice that Microsoft and IBM are sponsors? Notice that Oracle is not...

I would like to hear why Oracle should join (as a caveat I really have no idea what Oracle plans are in this area but assuming there is no immediate plan to join, I can help influence). In particular I would like to hear from people besides James tell me why (either leave a comment or send me a link to your blog if it's not on Planet Identity feed).

One of my side hobbies is studying how to give more effective presentations in particular with PowerPoint. Which to some of you I know seems like a lost cause but reality is that presentations to make or break many deals.

And one of my favorite blogs on this topic is Speaking about Presenting.

In a recent post they reminded their readers that when presenting there are 3 types of content we are delivering:

1. What you say
2. The slides you show
3. The handouts you give out

Personally I think the root cause of "death by PowerPoint" comes down to:

1 - The natural feeling that you have to share everything you know on a topic in your presentation (and worse) - your slide deck

2 - The belief your audience is interested in the same level of detail and worse - that they can absorb all of the detail just by listening to your presentation

3 - The (mis-guided) belief that a slide deck is an effective hand-out/post-presentation collateral

I will address thoughts on presentation "hand-outs" in another post.