Monday Feb 15, 2010

Directory Service Performance Optimization Strategy: Data Priming

Directory servers usually run for long period of times and have stable performances as all caches are warmed by the traffic. But how to get optimum performances as fast as possible right after starting the server ? Brad Diggs has published Directory Data Priming Strategies, another blog post added to the series of articles on Sun (now Oracle) Directory Server Enterprise Edition 7, ZFS and Flash Technologies.

Technorati Tags: , , , ,

Monday Feb 08, 2010

The basics of Flash Memory

These days, everybody get excited with Solid State Disks, flash memory and the performance improvements they have over other mass storage solutions.

Sun F20 Flash accelerator boardWe've been running some benchmarks of Sun Oracle Directory Server 7.0 leveraging new Sun flash based hardware modules. Before we go in details about their benefits, my colleague Brad Diggs posted a very educational article on the basics of Flash Memory to set a common understanding of the technology.

Read on and get ready for more data points on how ZFS and Flash Memory can improve Directory Server performances and scalability.

Technorati Tags: , , , ,

Friday Jan 22, 2010

Sun Directory compresses data for better performance !

Sun Directory Server Enterprise Edition 7.0 was released last November, and in the December timeframe Brad Diggs and Wajih Ahmed, both Principal Field Technologists and big experts in Directory Services, backed with engineers from the Directory engineering team and Mr Benchmark, put the product on the test bench to evaluate its performance and scalability with Sun new hardware and especially the new F-20 PCIe flash drives (see also what Mr Benchmark says about the F-20).

Brad's first article describes how much Directory Server 7 entry compression rocks, "extending search performance by more than 50% through increased caching potential". Brad provides details of his findings and gives the commands to run to get the benefits of DSEE 7 in your deployment.

The entry compression feature is also available in the technology that will power future versions of Sun Directory Enterprise Edition: the OpenDS project. In OpenDS, there are 2 options to reduce the size of entries stored in the database. The first one is called entry compaction, and it's enabled by default. The entry compaction feature removes all references to attribute names and replace them with small identifiers. The second option is actually entry compression which will use the popular ZLib algorithm. This option is not activated by default, but it's just a command away :

<OPENDS_HOME>/bin/dsconfig -X -p 4444 -h localhost -D cn=Directory\\ manager\\
 -w password -n set-backend-prop \\
 --backend-name userRoot --set entries-compressed:true

Below is the dsconfig usage for disabling entry compaction with OpenDS:

<OPENDS_HOME>/bin/dsconfig -X -p 4444 -h localhost -D cn=directory\\ manager\\
 -w password -n set-backend-prop \\
 --backend-name backend --set compact-encoding:false

Here's a table that compares the size of the databases of OpenDS 2.2.0 with no compat encoding, with it (default settings) and with compression enabled. The table compares the size of the entry record within the database as well as the overall size of the database which also includes indexes (default OpenDS settings).

Entry Count LDIF Entry Size Uncompacted Entry Size Compacted Entry Size Compressed Entry Size Uncompacted DB Size Compacted DB Size Compressed DB Size
100K 599 b 645 b 481 b 361 b 178.8 MB 163.20 MB 151.65 MB
-34% - 25% -9.6% - 7.1%
1M 603 b 649 b 485 b 364 b 1,515 MB 1,358 MB 1,243 MB
-34% - 25% -11.5% - 8.5%
10M 607 b 653 b 490 b 363 b 13,973 MB 12,416 MB 11,188 MB
-33% - 26% -12.5% - 9.9%

The percentages are computed from the reference value which is the default i.e. compacted. A negative value means an increased size, a positive one means a reduced size.

The second table compares the import times for the 3 different modes for storing entries, for the 3 sample data files.

Entry Count Uncompacted Compacted Compressed
100K 21 s 21 s 22 s
1.1% - -3.5%
1M 106 s 107 s 112 s
0.5% - -4.9%
10M 1006 s 1009 s 1101 s
0.2% - -8.9%

Note: in this table, negative numbers represent increase in time required to import compared to the default settings.

Enabling compression does result in a smaller disk use with that sample data (fully random values), but does come with a performance penalty at least at import time, less than 10% but the penalty increases with the amount of entries. If you've read Brad's article on DSEE entry compression, you understand that the smaller the entries in the database, the more can be potentially cached in the Database Cache and the better the overall performances are. So if your entries are quite large, contain values that are strings, you should consider enabling the entry compression with OpenDS.

Changing from the default mode (compacted) to uncompacted mode does not give any real advantage in performance, but does increase the disk space usage, so I do not see the value of changing these settings in OpenDS.

Anyway, the benefits of having compact entries in the database are available today with Sun Directory Server Enterprise Edition 7 and Sun OpenDS Standard Edition 2.2, and are helping customers to reduce the overall cost of ownership of the directory services.

Technorati Tags: , , , ,

Tuesday Oct 13, 2009

OpenDS 2.2.0 Release Candidate 1 is now available

Opends Logo TagThe OpenDS development team is very pleased to announce the immediate availability of OpenDS 2.2.0-RC1 which is the first release candidate for OpenDS 2.2.

OpenDS 2.2 offers the following new features from OpenDS 2.0 :

  • Scalable import and indexing
  • External changelog compliant with the Internet-Draft "Definition of an Object Class to Hold LDAP Change Records", draft-good-ldap-changelog-04.txt
  • Fractional replication
  • Extensible matching rules for time base attributes
  • Support for custom syntaxes based on substitution, regular expressions or enumeration
  • Remote server management in control panel
  • Recurrent tasks in control Panel
  • Default automatic Backup in the control panel
  • Separation of LDAP Servers and Replication Servers for replication
  • Ability to merge disjoint replication topologies
  • Dsconfig script friendly mode

We've also captured a first snapshot of the OpenDS 2.2 documentation and hosted it on it's own wiki: https://docs.opends.org/2.2/. The documentation is not complete yet, but will be almost at the same time we will do the final release of OpenDS 2.2.

The purpose of the Release Candidate is to solicit one last round of testing before the final release. So please test the OpenDS release with your client applications, in your environment or on your favorite platform.

If you do find a bug, please report it with Issue Tracker.

We welcome feedback. Please report you experience with OpenDS on our mailing lists, or on #opends IRC channel on Freenode.

OpenDS 2.2.0-RC1 is built from revision 5941 of our source tree.

The direct link to download the core server is: http://www.opends.org/promoted-builds/2.2.0-RC1/OpenDS-2.2.0-RC1.zip

The direct link to download the DSML gateway is: http://www.opends.org/promoted-builds/2.2.0-RC1/OpenDS-2.2.0-RC1-DSML.war

We have also updated the archive that may be used to install OpenDS via Java Web Start. You may launch that using the URL http://www.opends.org/promoted-builds/2.2.0-RC1/install/QuickSetup.jnlp, or visit https://www.opends.org/wiki/page/OverviewOfTheQuickSetupTool for more information.

Detailed information about this build is available at http://www.opends.org/promoted-builds/2.2.0-RC1, including the detailed change log

Major changes incorporated since OpenDS 2.1.0-build002 include:

  • Revisions 5870, 5888 (Issue #4181) - Resolves a Null pointer exception in DSML Gateway with specific substring search filters
  • Revision 5871 (Issue #4217) - Fixes an issue with ACI containing parenthesis in the description field
  • Revision 5874 - Improves the rebuild-index processing for performances
  • Revision 5880 (Issue #4252) - Fixes a replication issue between OpenDS 2.1/2.2 and OpenDS 2.0
  • Revision 5883 (Issue #4203) - Fixes an issue where restore -l (list the available backups) would exit with return code 1
  • Revision 5926 (Issue #4257) - Fixes an error raised when deleting recurrent tasks

Technorati Tags: , , , , , , ,

Monday Jun 29, 2009

FISL 10 Trip report

I've just spent a wonderful week in Porto Alegre, Brazil where I've landed to talk about OpenDS at the FISL 10 conference.
This is my first visit in Brazil and I must say that I didn't get any good impression of the country in the first two days. As a matter of fact, I didn't get any impression at all. I arrived on Monday evening around 9pm, it was all dark. After more than 16 hours of traveling, I just wanted to hit a bed.
On the Tuesday morning, thanks to the jet lag, I got up quite early, checked email and went for breakfast by 7am, noticing a rainy day and still pretty dark. I was just done with the breakfast when Bruno Souza arrived and took me to the location of the Javali meeting, an ancillary event of FISL, sponsored by Sun and organized by SOU Java and RS JUG.
We spent the whole day in the conference room, watching from time to time through the windows the heavy rain and wind. The Javali talks ended with pizzas and guarana and by then the night was already dark.
While I didn't get to see how Porto Alegre looks like in the first days of my visit, I did enjoyed the friendliness of Brazilians. At Javali, trying to follow the presentations in Portuguese was though but I think I got probably 50% of the technical parts thanks to the mix of english words and to my understanding of Spanish. And when it was necessary, Bruno or Mauricio Leal would do some translation for us.
I didn't get to talk at Javali, the agenda was pretty full and I hadn't told Bruno I would be coming as I wasn't sure I could make it. But Pat Patterson presented Securing RESTful Web Services with OpenSSO (and OAuth) and mentioned a few times OpenDS.

LP0_1036

LP0_1039Wednesday was the first day of FISL and all the Sun participants went quite early to help setting up the booth in the Exhibition Hall. Sun's booth was very well located and its main attraction was the thousands of small soccer balls that were given to attendees that registered to the OSUM program. I think that throughout the whole event, the Sun's booth was the most vibrant and busy one, with Roger Brinkley making demos with his toys, Angel Camacho, Brian Leonard, Kirthankar Das and others helping with installs of OpenSolaris on attendees' laptops.

LP0_1167LP0_1181

Arun Gupta fired the event on Wednesday morning with his presentation demonstrating the combined power of GlassFish, MySQL and NetBeans to build web applications.

Arun Gupta, inauguring FISL conf with the 1st talk

Friday was the busiest day for me as I was scheduled for 2 presentations. But before that, I was invited to participate in Simon Phipps talk show, describing in 5 minutes, what was OpenDS, what were the benefits for the Brazilian open source users and developers.

Fisl10 Simontalk

Immediately after, and in the same room, I did my presentation for OpenDS with the theme of "Scaling the Identity Store with OpenDS". The sessions talked about the 3 models we have in OpenDS for deployment :

  • Embedded in Java applications,
  • Standalone replicated servers,
  • LDAP Front-end access to MySQL Cluster's network DB.

While FISL is mostly attended by students, my session had a majority of System Administrators, interested by simplifying and reducing the cost of their data-centers.

4791_116007741662_583231662_2881035_1391095_n

Later in the afternoon, I was presenting again, repeating JavaOne's presentation from Tony Printezis and Charlie Hunt GC Tuning In the HotSpot Java Virtual Machine. Charlie was meant to attend the event, but the week before found out he could not make it. As they recalled I was in the room at JavaOne and I'm quite familiar with the subject as we're spending a lot of time trying the different options to tune the JVM to get the best performances out of OpenDS, they asked me to cover the talk. I think I've done a reasonable job, despite the density of information in the slides, and the simultaneous translation in Portuguese for the largest part of the crowd not so familiar with English.

Still on Friday, part of the exhibition floor was closed to the public as the Brazilian President, Lula Da Silva, was schedule to visit the event. Sun booth was very well positioned, on the border on the closed area and the crowd started to gather by the booth as President Lula arrived. The excitement was amazing. When the President reached by the OpenSolaris Brazil user group, he received an OpenSolaris cap and T-shirt from Vitorio Sassi, Sun employee and one of the leaders of the Brazilian OpenSolaris community.

Brazilian Presidente Lula with OpenSolaris community
Photo taken by Ludovic Poitou, June 26 2009.
Somerights20

.

On Saturday and last day of the FISL conference, I got to share a little bit more of the stage by answering a performance related question from the attendance on Bruno Souza's session about the future of Java,with the exceptional presence of Javali, the mascote for the Javali user group.

Bruno Souza with Duke and Javali

Overall FISL has been an amazing experience. It is definitely the biggest open source I've participated to. Over 8200 registered visitors, from 27 different countries, more than 320 speakers for 354 presentations and a presidential visit. More than that, Brazilians are extremely nice, generous and happy to live. They made our stay in Porto Alegre something that I'll remember for a long time. A special thanks to the main organizers: Bruno Souza and Eduardo Lima (here below with Simon Phipps)

LP0_1127

.

I'll definitely participate to the Call For Presentation next year, if evangelism of the OpenDS project is still one of my tasks for next year.

You can find all photos for the event in the FISL 10 picasa album.

Technorati Tags: , , , , , , , ,

Saturday Jun 27, 2009

To the FISL attendees...

FISL 10

Many of you have requested the slides.
Here they are :

Thanks for your presence...
A more detailed article is in the works.

Technorati Tags: , , , , ,

Tuesday Oct 28, 2008

OpenDS LDAP server under accidental stress !

Full speedBill Hathaway, long time user of Sun Directory Server product, was recently playing with OpenDS, the open source LDAP directory server written in Java. After installing and populating the server with a million user entries, he started a "modrate" job and forgot about it... Approximately 327 millions changes later, and after 13 hours, the server was still running fine at an average of 7000 modifications per seconds !

Read the details on Bill's blog.

Technorati Tags: , , , , ,

About

This is the blog of a senior software engineer, specialized in LDAP, Directory Server and OpenDS. Ludovic Poitou works in France at the Grenoble Engineering Center, in the Directory Services Engineering team. Outside work, I love skiing and taking photo

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today