Introducing the OpenDS Directory Service

The Sun Java System Directory Server has a distinguished heritage and a proven track record, with thousands of customers and billions of entries deployed. However the codebase is over ten years old and its origins are from a time when performance, scalability, and feature set requirements then were very different from what we're seeing today and expect to see in the future. It has also grown quite complex, and making a change to one area of the code may require an in-depth understanding of several other components. We're still working on improving this code, and the upcoming Directory Server 6.0 release will be the best we've had yet, but we are also preparing for the future and we think that an open development model needs to be a big part of that.

Last year, we decided to start from scratch, designing a new server from the ground up. Drawing from our years of experience, customer feedback, and some of our own ideas, we began working on what we hope will become even more enduring and successful than our current Directory Server. We're calling the new codebase OpenDS, and last Friday we rather quietly pushed the code out to https://opends.dev.java.net/.

OpenDS is an open source Directory Service written entirely in Java. I say "Directory Service" because we will include more than just the core LDAP-accessible database. Much like our current Directory Server Enterprise Edition, we'll also include directory proxy functionality (including virtual directory and data distribution capabilities), the ability to synchronize with Active Directory and potentially other sources, and various client-side tools.

To date, our development has primarily focused on the Directory Server itself, and we have basic support for all core LDAPv3 operations, and several extensions like controls, extended operations, and SASL mechanisms. However there are still a number of components to be added, like the access control subsystem, virtual attribute capabilities, and administrative interfaces, and there is significant work to be done in other areas like password policy and logging. We very much wish to have an open development process, and we welcome community participation. You can provide comments and feedback, file bugs and enhancement requests, participate in mailing lists, or even write code.

I'm sure that there will be a lot of questions about OpenDS, and our FAQ (I suppose in this case, that's "Frequently Anticipated Questions") at https://opends.dev.java.net/public/docs/OpenDS-FAQ.html aims to address many of them. If you still have other questions, then check out other parts of the site like our Documentation Depot or join our mailing lists. We'd love to have you help us achieve our goal of making OpenDS better than any other directory product out there.

Comments:

3 words - Apache Directory Server

Yes, yes, I've read the FAQ, and I'm sure the team that developed OpenDS had the best intentions, but if Sun really wanted to show their commitment to the open source community, and wanted to promote Java, they should have contributed to Apache DS.

And considering that Apache DS has been around for a long time now, and has not gained significant traction in the open source community, with what hubris does Sun think OpenDS will succeed wildly where Apache DS has not?

Posted by Rich Megginson on August 01, 2006 at 10:55 AM CDT #

Hi guys, and welcome in the world of OSS !!! Having a second Java OSS Ldap server is certainly a good news. I hope that we will be able to colaborate. Just to give a few reason why Apache DS didn't get used a lot those last year : we are not yet on a 1.0 release, and I just understand that people are a little bit reluctant about using a brand new product for critical tasks (and to be honest, they were right :) We are currently in the process of trying to deliver a 1.0 version which will be reasonably fast, seriously solid, and definitively open source !!!

Posted by Emmanuel Lécharny on August 01, 2006 at 11:07 AM CDT #

There are several reasons that we decided against contributing to an existing open source project like the Apache, OpenLDAP, or Fedora directories. Many of them are outlined in the FAQ, but I'll include them here for clarity:
  • The license model used by the Apache Directory Project is (as one would expect) the Apache License. This creates two problems for us. The first is that the Apache License is a BSD-style license, which does not guarantee that the code will always remain open. It would be possible for someone to take it and make changes without giving back to the community, and that is not something that we wanted to see. The second is that the Apache License could also be an inhibitor to the technology that we wanted to use. For example, they decided that they could not use the Berkeley DB Java Edition and had to go with something that (at least from a cursory glance) appears to be much more experimental and unproven.

  • Starting from scratch allows us to build a server from the ground up that we can ensure meets the needs of our existing customer base, is able to meet our performance and scalability requirements, and can provide a solid foundation for development we know needs to happen in the future. I've intentionally avoided looking at the codebase for the Apache Directory and other open source projects and have no reason to suspect anything other than great quality, but they don't appear to be aimed at the same types of users and deployments that we know we need to support.

  • It would be a very difficult challenge from a political perspective. We certainly would not want to appear as if we were taking over an established project, nor would anyone else want that. Further, if we found that our goals were not in-line with those of the established project then we could be forced with the decision to either not be in a position to support our customers, to drop out of the open source community, or to fork the existing code base, and none of those is a desirable outcome.

OpenDS is an entirely open project. All of the code is publicly available, and all of our development will be done in the open. Our issue tracker contains all of the features that we're currently targeting to include. We welcome participation from anyone who's willing to join us. However, we do have a customer base to support and we will not abandon them or put ourselves into a position that is less suited to meeting their needs. That is certainly not incompatible with open source or open development, but it would be extremely difficult to accomplish in an environment in which we were not involved from the ground up.

I think that it remains to be seen whether the Apache Directory will be a success as they are still fairly early in their life. I wish them all the best and hope that we can work together wherever it makes sense to do so.

Posted by Neil Wilson on August 01, 2006 at 11:55 AM CDT #

This is good news! FYI, Both ApacheDS and OpenLDAP projects currently have virtualization and synchronization capabilities through an open source project called Penrose (http://penrose.safehaus.org).

Posted by Jim Yang on August 01, 2006 at 12:30 PM CDT #

Well, after having read Neil post, I noticed some points that need a clarification : - There is no way a product already release under a ASL 2.0 licence could be removed of have its licence changed. Of course, a later version coule be licenced under 3.0, but the previous one will still remained under ASL 2.0. The licence will \*always\* allow you to use the code if you respect the condition of the licences, which are quite light. As Apache foundation is not owned by a private company, but supported by individual, I don't see any possible way that this fact change before you and I die :) - The ASL licence is written in a way that it's perfectly legal to use the code, modify it and ambed it in a commercial product, as soon as you just avoid to remove the licence in the file headers, to replace Apache name by any other name (appropriation is forbidden), and to use Apache at your own advantage. IBM does not have any problem selling IHS which is Apache HTTP server. (FYI : http://www.apache.org/foundation/licence-FAQ.html) - ASL licence will be an inhibitor only if you decide to respect other licence restrictions. The exemple you mentionned is really bdaly chosen. Even you you wish to use BerkelyDB JE, then without boing a licence to oracle, you will have \*no way\* to include BDB JE into your CDDL licence (unless Oracle buy Sun, or the opposite, but this is less likely, IMHO ;) If you want to embed BDB JE, what you will have to do is to ask your users to buy a licence. - Starting from scratch may seems a good idea, but it may also be seen as a NIH syndrom. (Not Invented Here). Of course, there are some problem with existing software and code, mainly this IP madness. We, at Apache, decided not to look at code which is not ASL or ASL compatible. That makes sense. But in the contrary, avoiding to look at existing code in order to avoid "spoiling your ideas" seems just insane. Trust me, it's really painfull to reinvent the wheel, and if we do so, it's just because of legal issues. The targeted users is totally off topic. As far as I know, apache products are widely used, so I guess we just target the very same user base. - Political aspects are somehow well managed into the Apache foundation. there are rules that are followed, like we don't accept a Project Manager who will have the final decision. We vote. We believe in Democracy, and in Meritocraty. Nothing lese. At least, the Project Manager is not someone who is directly linked to a unique private company. It can be anyone. Really. I understand that for Sun, this is a very difficult point, and that it's almost impossible to accept, from the political point of view. I accept that, but then, is it really Open Source? Or is it OpenSource but Private Company Decisions? Further more, if your concern is that the common decision does not fulfill your requirements, then you have the right to get the sources, and to fork. The point with ASF is that we are driven by a general consensus, which is to deligver the best product possible, for the largest possible user base. Short term decisions, user's driven decisions, are sometime really bad. We are fighting to avoid them, and it's easy because we don't have any pressure from a Private Company, which has named a Product Leader to rule the project. Ok, to be honest, those points have already been discussed extensivly since Sun decided to create the CDDL, and certainly in a better english :) What I just want to say is : guys, you are welcome, but nobody on earth will ever force you to join us. This will be each one private and personnal decision. Last thing : becoming an Apache committer is a process which is based on merit. You don't ask for this status. You deserve it. What you call Project Owner selection is totally opaque, while PMC members are voted. This make things easier for the project to have such an open process. We don't have sheperd, we have committers that decide, depending on the time they have, and their competences, if they can deal with contributions. Contributors can be anybody : no need to sign any paper, to send any request. You just drop a piece of code, just being informed that you grant licence to ASF for inclusion in the project. The committer will be responsible to check that the code is ok (technically AND legally). This may seems dangerous, but it works, it's simple, and its powerfull, because it empowers people. You fill part of something, a citizen of a world without frontiers, but with clear rules. Call me naive. But I like the way Apache works. I really want to share this feeling. Nothing more. A last word : success does not mean that 10 millions of people use ApacheDS. We are proud enough to be able to release a product that simply work, and that is already being use in some other project like Geronimo. This is the kind of achievement we are targeting. OpenLdap is really very successfull, and it deserves it. We hope to have one tenth of OpenLdap users in the next three years, this will be a huge satisfaction for us. And even if we only have ONE satisfied user, this is enough. Longue Vie à Open DS :)

Posted by Emmanuel Lécharny on August 01, 2006 at 01:45 PM CDT #

(Sorry for the really bad formating of te previous post... It's 4am here in Paris, and I'm really tired :)

Posted by emmanuel Lécharny on August 01, 2006 at 01:47 PM CDT #

Thanks for your clarification, Emmanuel. You certainly added a lot of valid points, and there were a lot of things that I left out in my initial response (mainly because I was trying to keep it reasonably short), but I definitely didn't want to imply that there's anything wrong with the Apache License or the processes in place for the Apache Directory. We just didn't feel like it was appropriate for us. We did put a lot of thought into our governance model, and Virtually all of our processes were based on those of existing successful open source projects.

The only thing I wanted to add was a note on our use of the Berkeley DB Java Edition. First, let me say that we are NOT open sourcing Berkeley DB JE, because Sleepycat did that under their own license. We're merely using it under its existing open source license, and that use is completely compatible with both the Sleepycat License and the CDDL. Anyone that wants to use OpenDS can use our backend based on the Berkeley DB Java Edition as a part of it without any cost or restrictions (if they want official support for JE, then they can of course buy it from Oracle). For any commercial release based on the OpenDS source that we may have in the future, we have already purchased the appropriate license and support contract, so we and our customers are covered.

Posted by Neil Wilson on August 01, 2006 at 02:20 PM CDT #

Hi Neil,

yes, for sure, this licence issue is a complicated one. I just think that when Sun decided to introduce a new one - for reason I really don't get - was just making things a little bit more obscure from a user point of view. However, that's life. Darwin's law will work out if it was a good move :)

Regarding BDB JE, the real problem is that we can't deliver a server which will be based on it. We can't tell our user "guys, BDB JE is OK, we are bound to it, but, sorry, you have to download and install it by yourself", at least for the very first version. That does not mean at all we are not considering using it. The backend API we have wrote perfectly address this problem : we wanted to be able to switch from JDBM to any other backend, for many reasons, like reliablity, speed, functions (transactions, backup, failover, DB replication etc). JDBM is just our RI. So I guess that at the end the solution chosen by both projects - OpenDS and ApacheDS - will be the very same : an abstraction layer above a storage system. We didn't invented it : OpenLdap already implemented it :)

Neil, I'm more than happy to see that Sun is coming with a java based Ldap server : it shows that DS are again gaining some traction. Alex Karasulu and the other fellows who started to write ApacheDS 4 years ago were very insightful. </br> I'm sure that as we have now 4 ldap servers which are clearly open source, or at least free to use, we will be able to see an emulation and a improvement of each one.

And, at the end, this will be great fun :)

PS: why don't you use something like http://www.fckeditor.net/ ? Formating text is really difficult in a stupid HTML textbox :)

Posted by emmanuel Lécharny on August 01, 2006 at 06:46 PM CDT #

[Trackback] いま cn=Directory Managerのエントリ 読んで知ったんだけど, OpenDS プロジェクト が java.net にて先週から始まっている. OpenDS は完全に Java で書かれたオープンソースのディレクトリ・サービスです. なぜ 「ディレクトリ・サービス」 という言葉を使うかというと, わたしたちは単なる 「LDAP アクセス可能なデータベース」 以上の機能をこの中に盛り込...

Posted by tkudo's weblog on August 01, 2006 at 08:43 PM CDT #

All this praise... But has anyone thought that taking a WONDERFUL product that's written in C and replacing it with some piece of java junk is a really good idea? I mean, I use Java in certain places in my infrastructure -- happily -- but I'm not certain that I'd want it as my directory service. Sun "Java" Directory Server has been amazingly \*reliable\*. Reliability is what directory services are all about. It pains me to say that I'm going to have to be looking at OpenLDAP now...

Posted by Robert Banz on August 02, 2006 at 01:56 AM CDT #

We strongly believe that writing OpenDS in Java will allow it to be even more reliable than our current Sun Java System Directory Server. The exception handling capabilities provided by the JVM are phenomenal, and it is relatively simple to keep any failures localized (and if an exception does occur, then you can even have it tell you the line number in the source code that was executing). It's been quite a while since I've seen any kind of failure in the OpenDS codebase that would cause the server to crash or render it unusable.

Another great benefit that Java offers over C is a tremendous amount of debuggability. Java makes it possible to get a great insight into what's actually going on, both from inside and outside the JVM. And it's allowed us to build a significant number of such features into the server. For example, you can get a stack trace from the server any time you want by retrieving the "cn=JVM Stack Trace,cn=monitor" entry, or by interacting with the server over JMX. We've also embedded a simple profiler in the server, so that if it's running more slowly than you would expect you can enable the profiler for a few seconds to capture information about what's happening (without needing to restart, and without significantly degrading performance any further).

Finally, we're adding a ton of reliability and serviceability features into the server that aren't directly tied to the Java platform. We've got a monitoring and alerting framework, where the server can let you know (through a pluggable mechanism like e-mail, JMX notifications, or SNMP traps) that there's a problem. We also intend to provide much better integration with existing monitoring frameworks like SMC, Big Brother, Nagios, MRTG, and HP OpenView.

We have no intention of developing something that is slow or unreliable or in any other way inferior to our current server. We want to be better in every way, and we've got a great start to that.

Posted by Neil Wilson on August 02, 2006 at 02:36 AM CDT #

Robert, Before starting looking at OpenLDAP, I would suggest that you wait until Sun Java System Directory Server 6.0 is out. This is just a few months away. The Directory Server 6 product line is built on the C code, and still has a long life in front of it. We're working hard to improve it especially stability and reliability, and I'm sure you will be amazed by the new features. We're continuing to invest in it. But Neil's right, there is a point where a rewrite is absolutely necessary to take a big step forward, and I truely think that Java is the best choice moving forward.

Posted by Ludo on August 02, 2006 at 04:48 AM CDT #

Post a Comment:
Comments are closed for this entry.
About

cn_equals_directory_manager

Search

Top Tags
Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today