Debugging Puppetmasterd from Puppet in Solaris 11 Express (and others)
By Michael Tin on Dec 07, 2010
I am evaluating Puppet for possible use in our labs. For those of you who do not know of Puppet, it is a Ruby based automation tool in the same vein of CFEngine and other system automation tools. This comes in very handy in medium to large deployments to make sure site-wide configurations are guaranteed to be identical. Hand building and tweaking systems becomes tedious and prone to error once the number of systems grows to maintain consistency in the configuration details.
As I was going through the Puppet installation on my Solaris 11 Express workstation I noticed that puppetmasterd was not starting an instance of puppet. I had followed the instructions in the Configuration Guide from PuppetLabs for Puppet 2.6.4. I had installed from the Ruby source instead of using one of the many prebuilt packages available on the interwebs and package repositories.
One major thing that I found non-intuitive with puppet was the options that you can run with puppetmasterd to debug problems. This is the daemon that starts the puppetmaster service on the host listening on port 8140. When I ran it, the puppetmasterd output was clean and it returned me to the shell. Unfortunately after testing with my puppet client, it appeared that the service was not running. I confirmed this with the puppetmaster zone seeing nothing binding to port 8140 with netstat and not seeing the output when I ran ps -ef | grep puppet.
One critical thing that I did was follow the instructions to use puppetmasterd --mkusers which automates a bunch of the tasks like creating the puppet users and groups, making certificates, and much more.
My first reaction was to run puppetmasterd --help and look for options that might give me debug output. Well there is a debug flag (--debug) and also a verbose flag (-v). I tried running them both with puppetmasterd and I still didn't get anything. Then I redirected the output to a file since I wasn't logged in to the zone console on the puppetmaster thinking that the output was being sent to the console. No such luck either. After searching around on the interwebs, I finally found the debug output I was looking for. I was missing the trace option which prints the stack trace.
puppetmasterd --no-daemonize --verbose --debug --trace
Once I did that, I saw the following:
err: /File[/var/lib/puppet/rrd]/ensure: change from absent to directory failed: Could not set 'directory on ensure: Permission denied - /var/lib/puppet/rrd
Sure enough, when I checked the permissions of /var/lib/puppet, it was all owned by root:root.
Then, and only then did it seem to show me what I needed to do to fix the problem. Once I did a chown -R puppet:puppet /var/lib/puppet everything was happy! My main gripe was that the debug and verbose options should have been enough to see that error. I don't see why I have to print the stack trace to get the debug information I was looking for. The error did not manifest itself when I used debug and verbose flags, it only came out when I added the trace flag. I expected the trace flag to show me the stack traces, not to show me additional debug error messages that actually told me what the problem was!
I will have a future followup blog about how I used new features in
Solaris 11 from OpenSolaris such as Zones and Crossbow networking to
virtualize my network connection and run it through NAT on my
workstation for testing. I was able to have a puppet master and multiple puppet clients virtualized on a single machine connected through a virtual etherstub network courtesy of Crossbow.