Caution when installing new server certificates in topologies running Solaris8 and/or Solaris9 ldap client machines
By marcos on Aug 13, 2007
Imagine that a couple of years ago you succesfully deployed secure Directory services in your network: you installed a Directory server 5.2 Patch2 based server infrastructure, configured SSL, installed a cert7 server certificate and its corrsponding cert7 CA certificate in the DS certificate database, deployed solaris LDAP clients all around your network (a mix of solaris8, solaris9 and solaris10 clients), made the cert7 CA certificate available to all of those clients so that they could validate the DS server certificate upon SSL handshake connection establishments, etc. Imagine that such secured LDAP-based Naming Services succesfully run for more than a year without interruption. Imagine that you decided after that time to upgrade the level of your Directory Server instances to, say, the DS5.2Patch4 level. Still, your secured LDAP-based Naming Services continued to succesfully run after such upgrade forever and ever.... Until yesterday!
What happened yesterday? All of a sudden, all the ldap clients stopped binding to the DS instances. A fast inspection of the DS logs immediately provides the root cause: the server certificates have expired and they need to be renewed.
The certutil command line tool provides an easy way to execute such renewal:
This said, you prefer to take advantage of the DS Console "Manage Certificates" Wizard capabilities and decide to generate a request for a brand new certificate instead. The certificate is provided by your Certificate Manager Server and it is imported inside the DS5.2 Patch4 server along with its corresponding CA certificate. As the NSS (security library) level has changed between DS5.2Patch2 and DS5.2Patch4, the newly created certificate and the corresponding CA certificates will be stored in cert8 format, so you decide to spread the new cert8 databases into the client machines to make the new cert8 CA certificate available to all of those clients so that they could validate the DS server certificate upon SSL handshake connection establishments.
Once the new certificates are in place, you test your DS instances... No errors seem present in the logs... That's good news.
Then you start your solaris10 clients, which open SSL connections against the DS instances using the new server certificate installed.... It all works... That's even better news.
Then you decide to start your solaris8 and solaris9 ldap clients... And then is when the bad news start: such clients will not be able to establish any bind with the DS instances yet.
Why is that?
The root cause is inside the NSS (security library) level in such boxes: Solaris8 and Solaris9 NSS level only allows to open and use cert7 certificates, in other words, the newly deployed cert8 CA certificate can not be used to validate the DS server certificate.
What can we do to get out of this mess once hit?
Here are the set of options I can think of at the moment, orderly listed from "preferred" to "highly unlike":
Option 1: the safer way to proceed is to remove SSL for solaris 8 and solaris 9 boxes while we think in the interim how to migrate them to solaris 10 to have the same security level everywhere. To configure them to use Non-SSL connections, we'd need to modify the client profile entry by setting the authenticationmethod to "simple", then re-init the ldapclient using the following command:
ldapclient -i -v -b defaultSearchBase=$profile-root-dn -P <profile>
-D "$proxyagent-dn" -w <password for proxy account> <DS IPaddress>
This interim solution may be accepted by a big audience: while running naming services using SSL is a requirement in some very secured environment, running ldap naming services in non-secure mode provides you with the same "risks" as running the highly spread regular good old NIS with the additional advantages LDAP provides, specifically in terms of replication of updates - incremental as opposed to whole map updates-.
Option 2: remove the newly installed certificates and try to renew the expired ones. We know the expired ones were signed using a cert7 CA cert that is still somewhere in the cert DB, otherwise all these problems that we have now would have appeared before the expiration of the passwords. This option assumes the old CA cert is not expired yet.
Option 3: (only for the braves) Running the steps listed below.... good luck in such a case. The solution below is possibly the same one you run up until yesterday during the last 2 years condensed in 8 steps: indeed, if you think backwards there is a big chance that the 8 steps below will most likely reflect the evolution your topology has suffered since it was first installed: at day 1, there were only cert7 everywhere. Then you upgraded the DS and everything kept working because the clients were still using the old copies of the CA certs.... But today, with the install of brand new cert8 based certificates and most likely cert8 CA certificates, we have broken that ability:
- Install a new instance named foo on serverroot /foo using the DS5.2patch2 version
- Create a new server certificate, foo-serv-cert and sign it with an old-format cert7 CA certificate (foo-CA-cert)
- At this time we have foo-serv-cert and foo-CA-cert in cert7 mode both
- Backup the DB directory (foo-backup)
- Migrate DS5.2Patch2 to 5.2Patch4. The DB directory will be migrated to cert8 as well, foo-serv-cert and foo-CA-cert are now in cert8 mode both
- Extract foo-serv-cert and foo-CA-cert are in cert8 from that newly upgraded DB and import them into our aer and aether systems
- Get the cert7 files from the foo-backup in step 4 and place them into the client-side DB directory
- Make foo-serv-cert the new official server certificate inside your DS instances
Option 4: fix ldapclient on solaris8 and solaris9, pretty unlikely to be achieved because it is NSS depending, and we can't just upgrade the security level of the OS like that.