Thursday Sep 23, 2010

Highly Recommended Solaris Tools book

Ever since I've been given a copy of the "Solaris Performance and Tools" book I've found it more and more useful. It has a lot of good information regarding various diagnostic tools that are native to Solaris including dtrace and mdb. If you ever need to examine a core dump or kernel panic the mdb section of this book is fantastic. Same goes for using dtrace.

Monday Nov 03, 2008

How to configure advanced kadmind logging in Solaris

After some experimenting and looking at source I've determined that the kadmind does have support for rotating its own log that is separate from the krb5kdc log (by default the kadmind logs to the log used by krb5kdc). To configure this, edit /etc/krb5/krb5.conf and add:
                                                                                                   
        admin_server = FILE:/var/krb5/kadmin.log                                                                        
        admin_server_rotate = {                                                                                         
                period = 1d                                                                                             
                versions = 10                                                                                           
        }                                                                                                               
in the [logging] section. Unfortunately, this is not documented properly in the krb5.conf man page but it basically works the same as the kdc_rotate parameter which is documented in man krb5.conf.
Note, to configure both the kdc and kadmind logging behavior to log to separate files, use something like:
                                                                                    
[logging]                                                                                                               
# commenting out default so kadmind will log to a separate file                                                         
#        default = FILE:/var/krb5/kdc.log                                                                               
        kdc = FILE:/var/krb5/kdc.log                                                                                    
        kdc_rotate = {                                                                                                  
                                                                                                                        
# How often to rotate kdc.log. Logs will get rotated no more                                                            
# often than the period, and less often if the KDC is not used                                                          
# frequently.                                                                                                           
                                                                                                                        
                period = 1d                                                                                             
                                                                                                                        
# how many versions of kdc.log to keep around (kdc.log.0, kdc.log.1, ...)                                               
                                                                                                                        
                versions = 10                                                                                           
        }                                                                                                               
                                                                                                                        
# controls kadmind logging                                                                                              
        admin_server = FILE:/var/krb5/kadmin.log                                                                        
        admin_server_rotate = {                                                                                         
                period = 1d                                                                                             
                versions = 10                                                                                           
        }                                                                                                               
This is the supported way to rotate the krb5kdc and kadmind logs. Also note that the kdc.conf man page is in error regarding the logging section. Use krb5.conf to control KDC logging instead.

Thursday Jun 16, 2005

Everything You Wanted to Know About Kerberos Enctypes But ...


I wrote a presentation about Kerberos encryption types (enctypes) and how they are used in Kerberos. It is aimed at both developers and administrators. You can download the PDF version here . Note, earlier versions of the presentation had a Sun Confidential label on the bottom of the slides which was left there by mistake. I have removed this label in the latest version of the presentation. I've updated the presentation slightly as of Oct 8,2007.

Technorati Tag: Technorati Tag:

Tuesday Jun 14, 2005

Playing with Solaris memory debuggers

Playing with Solaris memory debuggers

Playing with Solaris memory debuggers

The following are notes that I've made for myself as I used various Solaris memory debugging libraries. Given the following was originally for my consumption I can not vouch for the correctness of the grammar.

====================================================================
UserSpace info:

I was playing with both watchmalloc.so.1, libumem.so.1 to see what they can do.
Here's what I observe:

Using environment variables:

LD_PRELOAD=watchmalloc.so.1 MALLOC_DEBUG=WATCH,RW

watchmalloc can find bugs like (using MALLOC_DEBUG=RW):

p=malloc(128);
free(p);
foo=\*p; /\* invalid read \*/

or:

p=malloc(128);
free(p);
\*p=1; /\* invalid write \*/

or:

p=malloc(1);
memset(p, 1, 10); /\* write past buffer \*/

Note, watchmalloc will core dump when it detects an error.  And it causes
programs to run MUCH slower.

===========================================================

libumem does have some memory guards to detect invalid writes but it requires
use of the mdb command ::umem_verify on the core.  It does not catch bad reads
like watchmalloc but it does have guards with patterns like 0xdeadbeef and
0xbaddcafe.  And it does not slow down program execution like watchmalloc.

Use LD_PRELOAD=libumem.so.1
With libumem one can do memory leak detection (using
UMEM_LOGGING=transaction UMEM_DEBUG=default):

p=malloc(128);
p=malloc(128);
abort();

then do 'echo ::findleaks|mdb core' to see: 

CACHE     LEAKED   BUFCTL CALLER
000bb088       1 000cb608 main+8
----------------------------------------------------------------------
   Total       1 buffer, 320 bytes

then use:

echo '000cb608::bufctl_audit' | mdb core

to see the stack trace where the leak allocation took place.
Note, LEAKED is the number of times a leak occurred.

Note, a core dump is not necessary.  Use "mdb -o nostop -p PID" where PID
is the proc. ID of the running process and then do the findleaks stuff:

echo '::findleaks' | mdb -o nostop -p $(pgrep gssd)
echo '000cb608::bufctl_audit' | mdb -o nostop -p $(pgrep gssd)

This is good for daemons.  Or use "gcore <PID>" to get a core dump of a
running process.  This is useful to look for leaks in daemons like krb5kdc.

Also, to watch the memory size of a running daemon to see if it is growing over
time use:

prstat -p <PID of daemon> 300 > /tmp/prstat.out
---------------------------------------------------------------------------
Both watchmalloc and libumem will core dump on double free()'s.

Based on the above it seems like it would be good to use watchmalloc for
some memory corruption testing and use libumem for memory leak
detection.

Note use:

print ::umem_status|mdb core

to see umem status for user space core when debugging with libumem.

Use: 

print ::umalog | mdb core

to see umem transaction log and stack traces.

It's also good do do:

print ::stack | mdb core

and look at the stack trace (note, the values in each function listed
are the input registers %i0-5).
Look at umalog to see if it's possible to determine if memory was
free'ed earlier.  Use:

print "<address>::dis" | mdb core

to see the assembly around the stack function address to see where the
call was made (look for other call's).

===========================================================
Kernel memory debugging info:

In /etc/system do:

    \* kmem lite flag, must use independently of other kmem flags
    \* set kmem_flags = 0x100
    \* kmem flags: audit, test, redzone
    set kmem_flags = 0x7

and reboot system.

Use:

echo "::dcmds" | mdb unix.0 vmcore.0

to see debugging commands (look for kmem stuff).

Note, the kernel kmem outputs debug messages to syslog.

Technorati Tag:
Technorati Tag:
Technorati Tag:
About

user12615206

Search

Categories
Archives
« March 2015
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    
       
Today