Clayton Donley's Blog

My 3-year old daughter is a big fan of band-aids. The band-aid in the picture is a particular favorite. For every real or perceived injury, a band-aid heals all.

Having delivered virtual directory software for quite some time, I've been asked to help deal with a lot of infrastructure wounds. Some of these are real issues that virtual directories can play a strategic role in solving. Others have been areas that are better suited for other identity management technology.

In these later cases, a virtual directory would have been a band-aid on a very serious problem, or maybe a band-aid on a problem that never existed in the first place.

My Favorite: My data sources are flakey.

This can be a real infrastructure issue, but it's rarely something a virtual directory is going to solve.

Why? Because you want your identity information fresh and secure. The virtual directory way of solving this problem would be with a cache. However, caches have several draw-backs:

1. They get old. Identity information needs to be fresh. Do you really want accounts hanging around for hours after someone has been escorted out of the building by security? Some vendors try to solve this by keeping remote repositories in sync, but this just opens up the virtualization layer to all of the problems of meta-directory.
2. They bypass security at the source. By using a cache, you are keeping a copy of the data in the virtual directory that will be shared by all users instead of understanding that different people may have different data available to them.
3. They're difficult to manage. A cache adds to the total cost of ownership of your infrastructure. After all, you'll need to back it up, secure it, replicate it, distribute it, and otherwise think about it a lot. By keeping the virtualization layer simpler, it can often sit in a closet without much attention at all.

What are the Alternatives?

In many cases, no alternative is really needed. That band-aid may have added some psychological comfort, but it wasn't what healed the bruise.

How badly does your data source really perform? Why is that performance acceptable today? Do you have a better place to get that information?

I hear the concern about network links a lot, but in these cases the concern often doesn't reflect the fact that if a link is down, people often lose access to the applications using the identity as well.

Often there is a bit of cover-your-rear here, and vendors can play a part. It's a bit like taking your car in for service and hearing "your breaks may be okay, but maybe you should replace them before you get into an accident". You're less likely to question the source of the advice if it sounds like they're keeping you safe.

However, you may in fact have real issues with your data sources.

In these cases, the answer is almost invariably that it's cheaper and better to fix the source. It's much easier to add an extra directory replica or replicate your database than it is to have a long-term solution that involves a product being successfully able to detect changes in all of your infrastructure components.

In cases where this data needs to move around, provisioning is a mature technology with a lot of success in solving exactly this problem.

Technorati Tags: virtual directory, identity management

We got a lot of response to our benchmark on the Web and in our email bag. The responses were by the press, customers, and (most vocally) our competitors.

Regardless of where the responses came in from, the one thing that was obvious: people were excited. Certainly nobody was asking if directories were dead, that's for sure. Dave Kearns actually commented specifically about how infrequently he sees things like this in directory.

Not surprisingly, the customer camp loved this benchmark, as it confirms what they have already known and experienced, while giving them a fairly specific set of instructions to duplicate this type of result. It also goes hand in hand with our investment in virtual directory technology to show them that we're serious about having the best-in-breed product in this space.

In fact, the report was SO detailed and complete that it was obvious that the competition was going to try to find ways to discredit the big picture by focusing on the minute details. Since we made no attempt to compare apples to oranges or obscure the way testing was done, we're absolutely confident that this is not only a very valid benchmark, but one that has been more scrutinized than any before it.

Now to address the complaints from the competition. These fell into basically 3 major categories:

1. This is too many entries. Nobody needs this many!

I addressed this to some degree in my original blog post on the topic. This is just something our high-end customers want.

2. This isn't enough entries. We can do more!

I found this interesting, particularly because of who was most vocal about it: Howard Chu from Symas. I find this interesting because a benchmark I'm always asked to comment on is this one, which he did against Sun.

So basically on a system with 8 cores and 16GB of memory in that benchmark was able to scale to 10m entries with some reasonable performance. Now I see that this report is from 2006, but there's nothing newer posted on their site. Specifically, nothing with great detail about a 5 billion user benchmark on a quad processor server. Their own 2006 report also shows that the server isn't exactly scaling in a very linear way as you add entries. If you look at our previous benchmarks, this is absolutely the case with OID.

Another note is that there's no indication in his University of Michigan mailing list posting about the size of entries or other factors in his benchmark. For example, the entry sizes in one of our earlier 100m benchmarks (also published in great detail) was 120 attributes. A lot of scale is very dependent on I/O and clearly a small entry is easier to scale than a large one.

That said, the key point being made with our benchmark was not that someone else couldn't do a bigger one, but that we could do it without making the wrong kind of trade-offs. We're the only directory that can scale into the billions while still taking advantage of key enterprise data management features, such as secure backups, transparent data encryption, database vault, and other things that come with building on mature data management technology.

In fact, the idea that Oracle Database doesn't scale is pretty funny in itself. While LDAP benchmarks are completely unstandardized and generally use some tools built by Sun (SLAMD, for example), database benchmarking is very standardized and Oracle Database is a clear leader in this area, not only in scalability, but price per TPC-C.

3. The benchmark isn't realistic enough

Jonathan Gershater from Sun's directory team has a very thoughtful blog posting about our results. Not quite as dramatic and feisty as Howard Chu, but certainly equally skeptical.

He basically has three points to make:

1. Turning off change logs helped our results

Yes, we did, but this is actually a best practice for most directories during data load, as you'd simply load the initial set on each of the main servers and then turn on replication rather than load the entries in replicas via LDAP replication. Changelog was disabled more to avoid a large accumulation of changes during the various repeated runs, but it may have helped incrementally in the 'modify' tests only.

2. Password policy disabled helped our results

Not quite. Password policy being disabled helped bulkload incrementally, no impact on any other LDAP operation result since only failed bind/compare operations and password updates take an incremental hit. These were turned off primarily to make sure we could use the entries being generated by the Sun benchmarking tool.

3. The queries generated weren't realistic

I disagree. In a 2 billion user environment, you're generally talking about groups of active users and groups of less active users (and even some inactive users). A better test would have been to make 10% of these requests completely random, which I understand a new SLAMD beta is able to generate.

Even then, if one were to cut the number of responses in half in order to make every single user truly random, the results remain outstanding with hundreds of millions of operations every hour.

In Summary

The benchmark is out there. It's a valid benchmark. Certainly people are going to do their own benchmarks and competitors will always find a flaw in any benchmark. If there's one thing we're looking to do, it's publish some of the typical benchmarks we do on every-day hardware as part of each release. And we're going to continue to do it in an above-board way with specific details as we did with this report in order to help our customers size and scale this software in the best possible way.

Technorati Tags: directory, identity management, LDAP

For those of you that don't like reading press releases and wonder what all he buzz is about our recently announced Service Oriented Security offerings, Tony Baer from OnStrategies has a great roll-up here.

Nishant Kaushik, one of our key architects, does a fantastic job of boiling down this announcement as follows:

SOS covers the four stages of an application lifecycle - development, deployment, administration and governance. With SOS, organizations can now centralize and externalize security solutions as part of a flexible security architecture. Recent identity related efforts like the Identity Governance Framework are also part of this architecture, providing the ability to deliver privacy-aware applications.

Certainly the key take-away for customers and those building everything from the next bank portal to the next Flickr is that you can get cohesive identity management as a service today, so you're better off crafting your value on top of these services rather than trying to do a better job with identity management fundamentals.

I'm in Seattle for my brother's wedding, so was unable to see Thomas' talk at RSA yesterday. I would love to get some email with first-hand accounts and feedback.

Technorati Tags: identity management, LDAP

AmTrust Bank packed the room at Oracle OpenWorld, but thankfully the web's a little larger and has more comfortable chairs.

There will be a Webcast on May 1st at 1pm EDT (10am PDT) that reprises the original session and includes some new material.

Follow this link to register for the webcast, which includes K.P. Singh and Peter Dinin of AmTrust and Forest Yin of Oracle talking about centralizing database accounts using the Oracle Database's Enterprise User Security (EUS) functionality together with Oracle Identity Management.

Technorati Tags:
directory, identity management, LDAP, security

Having just mentioned "The Cuckoo's Egg," I thought I'd share my first IT security experience. I started my career at a large enterprise on a team managing networks of servers and workstations from vendors like Sun, HP, Motorola, and the like. The events below took place in the early 90's.

User accounts were centralized using NIS (in some cases exported to files and distributed to individual machines) with home grown tools for doing everything from adding/removing user accounts to backing up servers.

Since we were a high-tech manufacturing company, we had many labs that contained specialized servers for testing. These specialized servers were generally wide open, with a large number of people holding privileged accounts (e.g. root). The lab machines were, of course, connected to the main network.

At the same time, many of the tools used on various servers required shared access, which was done through the use of group accounts. Since many of these tools were run by commands that would remote shell using that group account, it was typical for these accounts to allow direct access (i.e. without using commands like SU).

It should be pretty obvious after the last two paragraphs that we were set up for a train wreck. This train wreck was triggered by something unexpected:

Needless to say, someone at the company had apparently had an extremely bad experience with a dating service called "Heart to Heart". Rather than call the better business bureau or tell his friends to avoid the service, he (or she) decided to send everyone in the company an email with the simple phrase:

The email was sent using a group account on a Sun server running SunOS 4.0.3. The connection to that server was made from an open HP lab server. The connection to the open lab server came from another open lab server in another city and in another division.

All of the audit logs were enabled, but all of them simply logged that root or a group user had logged in and done some work. At no point was anything traceable to the user.

The result?

Because of the way the email was sent (large to lists, rather than bcc), large number of vacation mail messages were triggered that went back to the group account, which in fact had mail forwarding set up to the rather large group of people that had access to the account. This in turn triggered lots of other individual vacation mails, autoresponders, "bots", and so forth from every person on that list back to the same wide distribution list.

Within about 15 minutes, the entire email system was choking and it took hours to get things back to normal.

It could have been worse!

Ok, so technically the dating service itself didn't take out the network. We tightened things up significantly from that point on. I had no security responsibilities at the time and was not at fault, but the experience has stayed with me since.

If the person had been more upset with his or her employer than with a dating service, what untraceable havoc could have been caused? Probably a lot worse.

So I'll just leave this as a cautionary story to those of you who are in environments where only "the important systems" are under identity management. Lab servers, group accounts, and similar gaps reduce or remove accountability and can compromise the rest of your network.

Oh, and we can help. :-)

Technorati Tags:
identity management, security

So there's been a lot of press in the last few days about Facebook shutting off access from the recently announced Google Connect social platform. For those of you that pay attention to identity management, but haven't paid attention to this particular situation, note the following couple of quotes: Robert Scoble comments:

Facebook is being consistent here. Dave Morin told me a few months ago all about Facebook's concerns. Such as, what happens if you change your email address, will it change everywhere that your email address got copied to?

So when Robert Scoble wrote this evening that Google is in the wrong, I disagree. I think Facebook's intentions aren't to let users get data out of the network until Facebook is absolutely forced to do so, and then only on Facebook's terms (see Facebook Connect). The fact is, this isn't Facebook's data. It's my data. And if I give Google permission to do stuff with it, I'm damned well within my rights to do so. By blocking Google, Facebook has blocked ME. And that, frankly, kind of frustrates me.

Michael is clearly in the user centric identity camp with his comments.

Robert's comments (or rather his agreement with Dave Morin's comments) are very similar to the type of concern your typical IT manager has about moving data between applications...only on a much larger scale.

In an ideal world, your identity is your own and you have granular control over who gets what and where it moves.

In reality, if the average user was presented this question as often as identity was requested, it would be a bit like these "desktop firewalls" that continuously ask you to "accept outbound connection to IP 10.10.10.20". Meaning that you'll grow numb to the requests at some point and simply grow accustomed to clicking "Accept" for every request for that data, knowing that clicking "Deny" means that the program (or service) won't work.

This is actually one of the issues that must be overcome by the user-centric identity crowd. It's one thing to provide a layer of security and control, but another thing to make that meaningful in real life.

On the other hand, assuming a data owner (in this case Facebook) wants to actually share their information with a partner (in this case, Google), Robert's mentioned concern about keeping synchronized copies of the data is valid. What's the solution to this part of the problem (assuming that the real problem here isn't Facebook being concerned about Google cannibalizing their business by building a better widget with their users)?

Seems like a case for some combination of virtualization and federation. What do you think?

UPDATE: Note a slight update on Scoble's blog that clarifies a few key details. Removes some of the synchronization/staleness issue, but interesting never-the-less.

UPDATE2: Google employee Kevin Marks says I'm wrong in comments here. Here's his correction to this post: "Robert, you're wrong about Friend Connect data getting stale. It's fetched directly from your linked Friend Data sources, including other Social Networks, with short-term caching on Friend Connect servers. There is a live two-way connection - Friend Connect posts back events to the Social Networks' activity streams when the user choses to do so."

Technorati Tags: facebook, identity management, privacy, identity

Finally getting around to participating in the latest stream of blog postings following up the "meta-directory is dead" and "daddy, does Active Directory grow on trees?" discussions...

Nishant has already addressed some of these comments in his post from July 16. Mark has hit on other items in his post on the same day.

Now you just have to wait until Ian boils this down to a single sentence again and Dave Kearns finds me secretly agreeing with Kim Cameron on something and the discussion will have come full circle. :-)

James McGovern - July 13

James wants to know 5 things (paraphrasing and with my replies embedded):

1. Why shouldn't we all just put our identity eggs in Microsoft's basket since everyone already has some Microsoft?

[CLAYTON] If you consider that most companies also have Oracle databases and most of the information you'll be needing for fine grain entitlements (meaning the stuff beyond username/password) is stored there, shouldn't this question be why you're not putting your eggs in an Oracle basket?

[CLAYTON] Or better, yet, most of you are using some form of Oracle application (HR? CRM?) to master things like reporting structures, department-based groups, cost centers, who's purchased what product, and so forth. If we're going to pick de-facto standards based on existing deployments, why stop at the directory niche? This information is all coming online with web services and ultimately via identity services.

[CLAYTON] I'm using these examples to demonstrate that very little reusable enterprise information outside of username, email, and some groups are mastered in Active Directory. Sure, some people do use it for more, but it can't be counted on...

2. Are current provisioning products too dependent on central sources?

[CLAYTON] Not to my knowledge. I think it's the opposite. They assume that you don't have a central source...at least ours does.

3. Should virtual directory technology be embedded in new software or stand-alone?

[CLAYTON] We're doing both. We know that nobody will rewrite the old stuff, which needs to work in new identity environments. We also know that some vendors will just never get identity. On the other hand, with Oracle products the push is definitely to at least include a base level of virtualization to improve open-ness.

4. The ideal solution is for people to just write better apps and avoid using virtual directory.

[CLAYTON] Agree. I'd like my car to stop using gas, too. :-) Until that date when every app gets there, we've got virtual directory. We'll continue to publish our own best practices and tools via Liberty Alliance's IGF project and enable our own applications to take advantage of mixed environments.

5. Why aren't more people talking about CARML?

[CLAYTON] There's not been the kind of controversy that sometimes keeps things in the headlines. Quiet progress, if you will. VERY good and impressive progress, though. I think you'll start hearing more about this, though hard to tell if some of the more system-management focused vendors you mentioned will be at the forefront here. After all, most of them don't even have (or understand) virtual directory yet...

Jackson Shaw - July 15

I'll visit some of Jackson's other comments in another post, but wanted to address this part, which goes with James' question #5 above:

What's CARML? Can someone explain it to me? Certainly, until Gartner says it's important I won't be thinking about it... ;)

I'm very glad that Jackson puts his full and total faith in Gartner, because as we all know, the latest Identity Management Magic Quadrants look something like this:

Oracle -> Leader

Everyone Else -> Not So Much

Forester is pretty much in the same boat. So I guess you can all just make those checks payable to Oracle. :-) Joking aside, while I love a nice roll up, especially when they're in my favor, the truth is that things aren't always what they seem.

As I said, I'll drill into his specific comments in my next post.

Jeff Bohren - July 21

I'm in pretty awesome agreement with Jeff that the problem is in the apps that are out there today being account-centric vs. identity centric. Not to mention his experience with Active Directory deployments:

To answer the rhetorical question, the vast majority of AD deployments are not intended as identity stores (at least from my experience). In most enterprises AD is used to manage and control user access to Windows workstations, the intranet, email, and enterprise web applications. AD is not usually intended as a central repository of identity, although it often becomes that with varying degrees of success.

Of course, the hard question is how do you solve it, eh?

A few commendable vendors such as SAP support SAML, but it’s a very small list. Support for external identity services or other identity standards such as SPML and XACML is nearly non-existent.

Wow. Those are the most glowing words I've ever heard about SAP's efforts in the identity realm -- ever. Certainly not the kind of words I'm used to hearing from analysts. :-)

SPML certainly isn't a cure-all. XACML helps and we've got a strong product and even better strategy in this area, but it comes down to application adoption. This is certainly why we're building key integration with fine grain authorization into the platform stack as much as in stand-alone products.

Technorati Tags: directory, identity management, LDAP, meta-directory, virtual directory

If you had any doubts about Oracle as a player in the identity management space, this short article from Jon Oltsik on CNET summarizes the situation.

Once again, common wisdom was completely wrong. While others struggle or abandon this space, Oracle has vaulted to a leadership position. In fact, my sources tell me they see Oracle in every large deal these days.

I'll let you read the rest for his summary of how why this is the case...

Technorati Tags: identity management

Kim Cameron of Microsoft just reposted a shortened version of his laws of identity.

I really didn't comment much when these were first being developed, though I recall being at a number of forums and conferences where discussion about them took place.

While they've gotten a lot of focus at times, I have my doubts about how important and practical some of the "laws" are. Rather than just parrot the laws, I thought it would be useful to discuss these possible issues and see what others may have found that either mitigates these issues or bring them to focus.

Here are the shortened laws in bold and my take on them:

1. People using computers should be in control of giving out information about themselves, just as they are in the physical world.

There are two ways identity information gets populated and shared:

1. We put it there, or

2. Someone else put it there.

We can all control the first path. I can choose to fill out your web form or not based on whether I will exchange elements of my personal information for the value that you are providing.

For the second path, we use enterprise systems every day where the systems in use have some existing knowledge about us. Marketing databases are bought, built, and sold every day -- often by the same publications that will actively run shrill articles about how your privacy is being invaded at this very moment.

In effect, often times once you've done #1, it's hard to prevent #2. You can give a web site the technical ability to reduce #2 and actively enforce a stricter privacy policy, but the reality is that the Web 2.0 world is often driven by "free" content and services that will drive more, not less, of this collection and sharing.

2. The minimum information needed for the purpose at hand should be released, and only to those who need it. Details should be retained no longer than necesary.

Nothing wrong with this particular ideal.

For example, when I get mailings from third parties as a subscriber to Harvard Business Review or TheStreet.com, they are always sent by those entities, not directly by third parties -- or at least they appear to be.

In general, this is actually a good business practice. Oftentimes the data you're collecting has proprietary business value in itself. If your business has made the decision that you're willing to part with it, you're highly unlikely to worry that there's a law of identity related to this.

You might be a little worried if there is a REAL law related to this. It's not like hospitals can go around selling lists of patients to drug companies. This is where privacy laws come into play.

3. It should NOT be possible to automatically link up everything we do in all aspects of how we use the Internet. A single identifier that stitches everything up would have many unintended consequences.

So I guess I should stop using FriendFeed, Facebook, and LinkedIn, eh? :-)

Ok. I know that this isn't what's really being said here. What's really being said is that using a shared identifier across a large number of systems allows people to know things about you that they shouldn't.

True. That said, this is hard to do within an enterprise. Are we really on a path for convergence across the vast Internet?

4. We need choice in terms of who provides our identity information in different contexts.

All of the references to control remind me of how most Windows firewall products work.

Basically I click on an application or link and get a pop up window in the lower right corner of my screen that says something like this:

"Application XYZ is attempting to access the internet to connect to 192.168.1.5 on port 848. Would you like to allow this? YES/NO"

The one thing all users learn quickly is that if they click YES, the application works. If they click NO, it doesn't. After a while, the pop-up is just another annoyance for the user such the actual applications, hosts, and ports aren't even noticed.

Now translate this to most web applications where if I click YES, some amount of information is shared and I can access what I want. If I click NO, nothing is shared, but I can't get access. What do you think the typical user will do? Do users read the EULA and privacy terms before they click?

And keep in mind that if we automate the process of entering all of this information by keeping it on a electronic card, they'll actually notice even less about the information that they're sharing because they won't be entering it. It'll become Yet Another Dialog to Accept (YADA?).

5. The system must be built so we can understand how it works, make rational decisions and protect ourselves.

6. Devices through which we employ identity should offer people the same kinds of identity controls - just as car makers offer similar controls so we can all drive safely.

It's hard to disagree with these last two points. They're very attractive points and give the users a lot of control.

I do like things such as the new Firefox address bar, which actively help me figure out whether I landed where I intended:

I also like the auto-form fill-out functionality in most browsers that makes registering for the myriad of sites easier.

Combined, this lets me know that I'm sharing my information with the entity I think I am and can visibly see and adjust the information I'm willing to share.

What's missing here is user education. A year ago, you had to look at the link you were following and know the structure of a URL to understand that you were being phish'ed...or just not click on anything. Incremental enhancements, such as those in the address bar, give us a path towards training users to avoid these negative situations without requiring them to be geniuses.

After you've verified that the vendor in question isn't fraudulent in itself, the question becomes whether you want to give the information requested.

Portable identity is probably helpful here, but if I were an enterprise I'd be more focused on the back office.

Just as it's a bad waiter that stole your credit card number and not an evil plot by TGI Friday's, it's not the intent of most organizations to actively compromise private information.

The difference here is that instead of a handful of credit card numbers, we're talking about whole repositories of data.

This may be an identity management problem (e.g. user with too much access and not being audited), but it's just as likely to be a data management problem, backup security process problem, or other issues that can lead to massive insider compromise (accidental or intentional). If you're not solving these in a concerted way, it won't matter much what your privacy policy is except for any liability you've created for yourself.

In Summary...

Not saying that the laws of identity lack value in the real world. Not saying that users shouldn't control their destiny.

Am saying that we need to be careful to ensure that these laws line up with the reality of how people use computers and that the embodiment of the laws doesn't open up additional risks, while keeping us from focusing on systematic risks that might be taking place behind the browser in or applications, middleware, databases, directories, and back office systems.

Technorati Tags: identity management, privacy, security

There is a post today on Pingdom talking about the new Firefox SSL error page that appears when you try to connect to a site with a self-signed or invalid certificate.

As you see in the image above, it actually doesn't show the page you're going to until you explicitly allow it as an exception.

Pingdom goes on to talk about how this can create a lot of issues (particularly for internal sites), but then goes on to estimate that 18% of Fortune 1000 web sites would be affected my this.

Much of my comment on the laws of identity yesterday were related to the user experience and how we need to look at how users really use their computers and identity to understand the best real solutions to identity problems.

The question here is whether Firefox is over-warning. I would argue that it isn't. SSL with valid server certificates is one of the most basic steps a site can take towards being secure. Just because the US Army site above isn't using a valid cert and many other large companies neglected to update their certs doesn't mean that Firefox shouldn't be aggressive in its warning.

This is similar to the experience many of us had with white page directories in the 90's. At first the data in them was highly inaccurate, but once people started using them to find you or authentications were hooked into them, suddenly you couldn't work with inaccurate information and were motivated to fix the problem.

The same thing will happen here with these sites. Unless they want the millions of Firefox 3 users to be put off, they'll upgrade to this minimum level of security. Once they have, the exceptions will look particularly outstanding and be an instant red flag that a site might not be what it seems.

Technorati Tags:
security

Many of you know that I came to Oracle through the acquisition of OctetString. You may not realize that I co-founded OctetString in early 2001, which was during the last downturn. In fact, we were negotiating our first software sale when the 9/11 terrorist attacks occurred.

So I read with great interest Jason Calacanis's email (and blog post) discussing how startups can better survive an economic downturn. Given that he too started his last company (WebLogs, acquired by AOL--think Engadget) during the last downturn, he's got very solid advice. I don't agree with a few specific items (never had a reason to test dedication with Sunday morning meetings), but overall a great read.

I thought I'd share a bit of advice and a few tales from that same period of time, but in the enterprise software space.

When we first started OctetString and created what is now the Oracle Virtual Directory, we had a number of pre-baked customers that were lined up to buy our software. Unfortunately, most of these were telco customers and by mid-2001 our phone calls weren't simply finding people who had been pink-slipped, but entire divisions that had been abandoned and certainly weren't going to be buying software from us anytime soon.

What kept things going was pretty simple:

1. Keep costs low -- especially recurring

While all expenses should be reviewed, you're going to want to pay particular attention to things that are recurring, including people, rent, etc...

When we needed hardware, I just made a trip to the local liquidator. HP-UX server: $800. Solaris box: $995. A bit like going to a junk yard and not as glamorous as handing over a check to your local rep, but it works.

Until almost 2003 we didn't even have an office, and even then I just used a Regus facility in order to share some common services with other companies (and not worry about anything related to maintaining the office itself). Not to mention the lease was relatively short-term (and I loved their coffee machine).

2. Retain an insanely dedicated core group

Jason makes this point and mentions seeing who shows up for a Sunday morning meeting or such to see who's dedicated. I think you'll know who the right people are even without having to test them.

These people won't care about fancy offices or silly perks. My first sales guy closed a critical deal with Pfizer in the United economy class line at O'Hare Airport on the way to Germany. Another critical deal with Coca-cola was closed in a phone booth at University of Illinois while a kid was breaking up with his girlfriend in the next booth (it's not you, it's me). Not having an office or business class for international trips didn't seem to stand in the way of his performance.

Others were equally (and probably more) dedicated. In the earlier dry times it wasn't uncommon to be deferring paychecks.

3. Make something people actually NEED -- particularly during bad times

When we started the company, we noticed that most enterprises were still doing multi-year projects to consolidate and synchronize all of their user repositories with a technology called "meta-directory".

The underlying need was to get all user information into a single place for portals, ERP, HCM, CRM, and related business applications. These are big, important applications and every one of them needs information about usernames, passwords, roles, department numbers, reporting hierarchy, and so-forth to function.

This may seem trivial, and today using software like Oracle Identity Management it's a lot easier, but at the time it was a black art requiring lots of consultants, lots of software, and Big Dig style project timelines (and success rates, unfortunately).

We simply shrank many of these projects from years to days and the results couldn't be ignored.

One particular customer implementing a CRM solution with a lot of consultants estimated that they saved something like $10m in consulting over-runs alone.

You have to be having a huge impact that can't be ignored simply because you're not the right vendor. This is especially true when times are tight and customers become more conservative. Customers know that a lot of smaller vendors won't make it and don't want to be stuck with abandonware.

Looking forward to comments. Thinking to do a few more posts on this topic if there's any demand.

Got a lot of positive feedback on my post from yesterday about some lessons learned while growing OctetString, so will write a few more articles along these lines...

Given that consulting billing rates and hours go down in tough times, many consultants will undoubtedly decide to build businesses to "product-ize" some of their solutions.

This is completely possible -- our own Oracle Virtual Directory started out that way, as mentioned yesterday. Some of our best partners were started this way as well.

However, there are a number of common traps that consultants fall into when they enter the software business.

1. Repeatability, not complexity...

Repeatability in software gives you the ability to scale your customer base.

Consultants often work in the role of car mechanic -- look under the hood, find scary problems, suggest some solutions that will require parts and labor, and finally plan and implement that solution. Each customer has a new problem requiring different parts, plans, and execution.

Software is a very different business. You're looking for as much commonality as possible between customers so that what is delivered can be repeated at other customers with the minimum possible effort.

This doesn't mean that you can't solve complex problems or require services to implement. It simply means that the product-ized part of your product shouldn't be different for every customer.

It also goes without saying that building your software on standards-based middleware will help reduce the amount of post-sales time spent doing customized integration with each customer.

2. Don't design around one big customer...

Even companies that don't originate from consultants tend to fall into this trap a lot, but consultants do it almost every time because they tend to be solving a problem that they encountered at a particular customer.

It is wonderful finally finding a customer or prospect who will spend a significant time helping you understand their requirements. You should certainly listen to them -- they are the customer, eh?

The trick is to use that customer to validate your approach rather than try to solve every esoteric problem that the customer might have through your software.

You might consider providing extension points, allow for customer designed templates, and so forth to accommodate their needs without building less repeatable stuff into the product. You'll also want to consider that your product may not be the right place to solve a particular issue.

3. Consulting isn't Software Sales

Many consultants have great people skills. They can help customers understand complex technology as it relates to the customer's own environment. Customers often base decisions about technology purchases in-part on recommendations from expert consultants.

However, this doesn't always (or even often) translate into being able to actually sell software. Not that you can't learn to do so, but the process of selling is very different from the process of pitching a solution as a consultant.

As a consultant, you have high credibility in part due to your independence. As a vendor, that credibility is diluted to some degree, even when you're still trying to help the customer do the right thing to solve their problems.

The overall process goes far beyond what you say and how credible you are with a customer. You're going to need to educate yourself and bring in the right people to help you be successful.

What happens when you don't know what you're doing?

A (now) funny story from our very first sales call (with Fannie Mae, oddly enough) back in early 2001 before we hired our first sales person or took the time to better understand the sales process:

I was on the phone with the customer while another person from the company was acting as the "account manager". Nobody is in the same room. The customer's first question: "What does this solution cost?" Oops! We hadn't priced it yet and hadn't discussed how we would handle this question. After an uncomfortable silence, the customer's question was answered (after a flurry of background instant messaging).

Thankfully we got better at this with time, brought in people that had experience selling enterprise software, and things worked out well.

I should also point out that the very first customer we did end up selling to was BEA Systems. This is why a portion of Oracle Virtual Directory's original 1.0 release is actually embedded in every copy of WebLogic 7.0 and above.