Wednesday Oct 08, 2008

Not able to mount from Fedora Core 9

Helen Chao, a colleague who had never really used Linux, asked me to help configure a kernel. I asked why and she said she needed to test RDMA over NFSv4. It turns out that the stock 2.6.25 kernel with Fedora Core 9 already had the support in it. We followed the directions at the nfs-rdma.txt and were not able to get it running.

Helen (a great test engineer) proceeded to investigate from there and couldn't get a simple loopback or NFS mount to succeed.

So I exported the root to all hosts and went to work debugging this issue. A 'rpcinfo -p' on the server showed the expected registered services. The same call from a client failed, but a ping worked:

[th199096@jhereg ~]> rpcinfo -p pnfs-9-30
\^C
[th199096@jhereg ~]> rpcinfo -p pnfs-9-30
\^C
[th199096@jhereg ~]> sudo mount -o vers=3 pnfs-9-30:/ /mnt
\^C
[th199096@jhereg ~]> sudo mount -o vers=3 pnfs-9-30:/ /mnt
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
nfs mount: retrying: /mnt
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
\^C
[th199096@jhereg ~]> ping pnfs-9-30
pnfs-9-30 is alive

I thought that perhaps it was a firewall issue and disabled IPTABLES.

No luck and I knew the mount should succeed - I tried it with my home Core 8 box and an OpenSolaris server. It worked, but then again, that Linux box has been configured for ages. Long story short, I asked Chuck Lever for help.

His only suggestion was to turn off selinux or as he puts it:

Also disable selinux, just so your systems behave like normal Unix.

So I followed the directions I found here: How to Disable SELinux and now the mount works:

# mount -o vers=3 pnfs-9-30:/ /mnt
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
nfs mount: retrying: /mnt
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
nfs mount: /mnt: mounted OK
# 

Most of the help I found with google on the RPC messages wasn't informative. Either the suggestion was to turn off IPTABLES or there was no reply.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Sunday Feb 18, 2007

Linux sendmail not sending mail and no error messages

Can I get a big "Doh!" from the crowd? I'm trying to upgrade my domain server from Fedora Core 4 to Fedora Core 6. I want to isolate what I will need to change to go to Solaris. Everything is kinda going okay, network addresses did not change after a reboot.

But sendmail is queuing my outgoing mail and not logging anything. And it was telling me what was going wrong, but the verbage was just too weird.

Make my changes to sendmail.mc and make:

[root@adept mail]# make
WARNING: 'sendmail.mc' is modified. Please install package sendmail-cf to update your configuration.

This actually means do the following:

[tdh@adept doc]> sudo yum install sendmail-cf

I just couldn't parse it correctly. Here is how I found my "Doh!" moment:

The mail queues have entries: [root@adept mail]# mailq /var/spool/mqueue (4 requests) -----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------- l1J0feON002810\* 9 Sun Feb 18 18:41 <root@adept.internal.excfb.com> <tdh@sun.com> ...

Some testing:

[root@adept mail]# sendmail -v loghyr@loghyr.com kdjfjklfs . loghyr@loghyr.com... Connecting to [127.0.0.1] via relay... 220 adept.internal.excfb.com ESMTP Sendmail 8.13.8/8.13.8; Sun, 18 Feb 2007 18:52:31 -0600 >>> EHLO adept.internal.excfb.com 250-adept.internal.excfb.com Hello [127.0.0.1], pleased to meet you 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-8BITMIME 250-SIZE 250-DSN 250-ETRN 250-AUTH DIGEST-MD5 CRAM-MD5 250-DELIVERBY 250 HELP >>> MAIL From:<root@adept.internal.excfb.com> SIZE=10 AUTH=root@adept.internal.excfb.com 250 2.1.0 <root@adept.internal.excfb.com>... Sender ok >>> RCPT To:<loghyr@loghyr.com> >>> DATA 250 2.1.5 <loghyr@loghyr.com>... Recipient ok 354 Enter mail, end with "." on a line by itself >>> . 250 2.0.0 l1J0qV2d002864 Message accepted for delivery loghyr@loghyr.com... Sent (l1J0qV2d002864 Message accepted for delivery) Closing connection to [127.0.0.1] >>> QUIT 221 2.0.0 adept.internal.excfb.com closing connection

Note that it is talking to 127.0.0.1 and that is not right. What does the sendmail config files look like:

[root@adept mail]# ls -la send\*
-rw-r--r--   1 root root 58203 Feb 11 10:58 sendmail.cf
-rw-r--r--   1 root root  7257 Feb 18 17:29 sendmail.mc
-rw-r--r--   1 root root  7209 Feb 18 17:19 sendmail.mc.stock

Okay, that hasn't changed today.

[root@adept mail]# make
WARNING: 'sendmail.mc' is modified. Please install package sendmail-cf to update your configuration.

I then get the "Doh!" and install sendmail-cf as shown above!

[root@adept mail]# make
[root@adept mail]# ls -la send\*
-rw-r--r-- 1 root root 59161 Feb 18 18:54 sendmail.cf
-rw-r--r-- 1 root root 58203 Feb 11 10:58 sendmail.cf.bak
-rw-r--r-- 1 root root  7257 Feb 18 17:29 sendmail.mc
-rw-r--r-- 1 root root  7209 Feb 18 17:19 sendmail.mc.stock
[root@adept mail]# service sendmail restart
Shutting down sm-client:                                   [  OK  ]
Shutting down sendmail:                                    [  OK  ]
Starting sendmail:                                         [  OK  ]
Starting sm-client:                                        [  OK  ]

Still not delivering and I am suspicious of why is it trying to talk to domains directly:

l1J0o9AP002853       33 Sun Feb 18 18:50 <tdh@adept.internal.excfb.com>
                 (Deferred: Connection timed out with www.loghyr.com.)
                                         <loghyr@loghyr.com>

I have to send outgoing mail through cox.net. Look what I have in my sendmail.mc:

[root@adept mail]# grep cox.net sendmail.mc
dnl define(`SMART_HOST', `smtp.central.cox.net')dnl

Bzzt, fix it!

And that flushes a bunch of requests after a make and restart!


Originally posted on Kool Aid Served Daily
Copyright (C) 2007, Kool Aid Served Daily

Monday Feb 12, 2007

Trying to get a Kerberized NFSv4 server/client on a NSLU2

Normally I don't summarize what I'm about to write about, however, I think this entry is all over the place. But there is useful information in here, So, I'm trying to get first kerberos and then NFSv4 working on a NSLU2 running OpenSlug. In order to validate my results, I also try to get a Linux NFSv4 server up and running on one of my Shuttle SS51G boxes. I finally get that to work, but I have no luck on getting the NSLU2 working correctly as either a server or client.

I decided to try another Linux client to see if I could get the process streamlined:

[tdh@sandman ~]> kadmin -p tdh/admin
Couldn't open log file /var/krb5/kdc.log: Permission denied
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin:  addprinc -randkey nfs/mrbill.internal.excfb.com
WARNING: no policy specified for nfs/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "nfs/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM" created.
kadmin:  addprinc -randkey host/mrbill.internal.excfb.com
WARNING: no policy specified for host/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "host/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM" created.
kadmin:  ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal nfs/mrbill.internal.excfb.com
kadmin: No such file or directory while adding key to keytab

Okay, not only do I need to fix the above, I also need to fix not being able to add to /var/krb5/kdc.log. We can get the keytab generated with:

[tdh@sandman /export]> sudo chown tdh:staff keytabs/

And we see:

kadmin:  ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal nfs/mrbill.internal.excfb.com
Entry for principal nfs/mrbill.internal.excfb.com with kvno 4, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrbill.keytab.
kadmin:  ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal host/mrbill.internal.excfb.com
Entry for principal host/mrbill.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrbill.keytab.

Okay, the first thing to note is that mrbill is running OpenSlug:

root@mrbill:~# uname -a
Linux mrbill 2.6.16 #1 PREEMPT Fri Jun 9 07:34:31 PDT 2006 armv5teb unknown unknown GNU/Linux

We try to get the keytab:

root@mrbill:~# mount sandman:/export/keytabs /mnt/sandman/keytabs
mount: can't get address for sandman
root@mrbill:~# host sandman
-sh: host: not found

Why? Well it turns out that:

root@mrbill:~# cat /etc/resolv.conf
search mshome
nameserver 192.168.2.108
nameserver 182.168.2.1

I thought that the domain entered in the turnup init was for the CIFS domain. Easy enough to fix...

root@mrbill:~# cat /etc/resolv.conf
search internal.excfb.com
nameserver 192.168.2.108
nameserver 182.168.2.1
root@mrbill:~#  mount sandman:/export/keytabs /mnt/sandman/keytabs
root@mrbill:~# cd /etc
root@mrbill:/etc# cp /mnt/sandman/keytabs/mrbill.keytab krb5.keytab
cp: cannot open `/mnt/sandman/keytabs/mrbill.keytab' for reading: Permission denied

What now? (Permissions)

root@mrbill:/etc# ls -la /mnt/sandman/keytabs
total 9
drwxr-xr-x  2 tdh  uucp  512 Feb 12  2007 .
drwxr-xr-x  5 root root 4096 Feb 12 08:22 ..
-rw-r--r--  1 root root 1968 Feb 12 06:50 krb5.conf
-rw-------  1 tdh  uucp  161 Feb 12  2007 mrbill.keytab
-rw-r--r--  1 root root  155 Feb 12 06:48 mrx.keytab

Fix them up on the server and:

root@mrbill:/etc# cp /mnt/sandman/keytabs/mrbill.keytab krb5.keytab

We need to get a good copy of krb5.conf, idmapd.conf, and sysconfig/nfs. For now, we will leave idmapd.conf alone, to illustrate the NFSv4 mapid issue.

root@mrbill:/etc# scp mrx:/etc/krb5.conf .
root@mrbill:/etc# scp mrx:/etc/sysconfig/nfs sysconfig

Now this time I know kerberos is not installed:

root@mrbill:/# ls -la ./usr/kerberos/bin/kinit
ls: ./usr/kerberos/bin/kinit: No such file or directory

And we can easily add it:

root@mrbill:/# ipkg list | grep krb5
kernel-module-rpcsec-gss-krb5 - 2.6.16-r6.6 - rpcsec-gss-krb5 kernel module
root@mrbill:/# ipkg install kernel-module-rpcsec-gss-krb5
Installing kernel-module-rpcsec-gss-krb5 (2.6.16-r6.6) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/slugos-bag/cross/3.10-beta/kernel-module-rpcsec-gss-krb5_2.6.16-r6.6_ixp4xxbe.ipk
Installing kernel-module-auth-rpcgss (2.6.16-r6.6) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/slugos-bag/cross/3.10-beta/kernel-module-auth-rpcgss_2.6.16-r6.6_ixp4xxbe.ipk
Configuring kernel-module-auth-rpcgss
Configuring kernel-module-rpcsec-gss-krb5

Still not there for me:

root@mrbill:/# ls -la ./usr/kerberos/bin/kinit
ls: ./usr/kerberos/bin/kinit: No such file or directory
root@mrbill:/# find . -name kinit

My guess is that you can export with kerberos, you just can't mount it.

We should confirm that!

root@mrbill:~# mkdir /home/nfs4
root@mrbill:~# chmod 777 /home/nfs4
root@mrbill:~# cd /home/nfs4
root@mrbill:/home/nfs4# touch see_me
root@mrbill:/home/nfs4# chown tdh:10 see_me
root@mrbill:/home/nfs4# ls -la
total 8
drwxrwxrwx  2 root root 4096 Feb 12 09:00 .
drwxrwxr-x  8 root root 4096 Feb 12 09:00 ..
-rw-r--r--  1 tdh  uucp    0 Feb 12 09:00 see_me

And I try to add the export:

root@mrbill:/home/nfs4# more /etc/exports
/home/NFS4 172.16.0.0/16(rw,fsid=0,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
root@mrbill:/home/nfs4# cd ..
root@mrbill:/home# ls -la
total 32
drwxrwxr-x   8 root root  4096 Feb 12 09:00 .
drwxr-xr-x  18 root root  4096 Feb  5 22:44 ..
drwxrwxrwx   2 tdh  uucp  4096 Feb  5 23:03 NFS4
drwxrwxrwx   2 root root  4096 Feb 12 09:00 nfs4
drwxr-xr-x   2 root root  4096 Feb  5 22:53 nfsv2
drwxr-xr-x   2 root root  4096 Feb  5 22:53 nfsv3
drwxr-xr-x   2 root root  4096 Feb  5 22:53 nfsv4
lrwxrwxrwx   1 root root     7 Feb  5 22:26 root -> ../root
drwxr-xr-x   2 tdh  staff 4096 Feb  7 21:21 tdh
root@mrbill:/home#

Looks like /home/NFS4 was created for me, or I'm suffering from severe memory loss...

I could have done this last week, note the time stamp.

root@mrbill:/home# ls -la NFS4
total 8
drwxrwxrwx  2 tdh    uucp 4096 Feb  5 23:03 .
drwxrwxr-x  8 root   root 4096 Feb 12 09:00 ..
-rw-r--r--  1 200096 uucp    0 Feb  5 23:03 ut

Must be memory loss!

root@mrbill:/home# cd NFS4/
root@mrbill:/home/NFS4# touch see_me
root@mrbill:/home/NFS4# chown tdh:10 see_me
root@mrbill:/home/NFS4# ls -la
total 8
drwxrwxrwx  2 tdh    uucp 4096 Feb 12 09:03 .
drwxrwxr-x  8 root   root 4096 Feb 12 09:00 ..
-rw-r--r--  1 tdh    uucp    0 Feb 12 09:03 see_me
-rw-r--r--  1 200096 uucp    0 Feb  5 23:03 ut

And yes:

[tdh@mrx ipk]> showmount -e mrbill
Export list for mrbill:
/home/NFS4 172.16.0.0/16

I was in 172.16.0.0/16 space last week. Touch up the export and:

[tdh@mrx ipk]> showmount -e mrbill
Export list for mrbill:
/home/NFS4 192.168.2.0/24

Okay, I do the mount and I'll claim it gets done as nfsv3:

[tdh@mrx ipk]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4
[tdh@mrx ipk]> ls -la /mnt/mrbill/NFS4
total 8
drwxrwxrwx 2 tdh    wheel 4096 Feb 12 03:03 .
drwxr-xr-x 3 root   root  4096 Feb 12 11:08 ..
-rw-r--r-- 1 tdh    wheel    0 Feb 12 03:03 see_me
-rw-r--r-- 1 200096 wheel    0 Feb  5 17:03 ut

Why do I claim it is nfsv3? Because I suspect that the idmapping should be hosed. Can we verify this? Yes:

[tdh@mrx ipk]> sudo umount /mnt/mrbill/NFS4
[tdh@mrx ipk]> sudo mount -o vers=3 mrbill:/home/NFS4 /mnt/mrbill/NFS4
[tdh@mrx ipk]> ls -la /mnt/mrbill/NFS4
total 8
drwxrwxrwx 2 tdh    wheel 4096 Feb 12 03:03 .
drwxr-xr-x 3 root   root  4096 Feb 12 11:08 ..
-rw-r--r-- 1 tdh    wheel    0 Feb 12 03:03 see_me
-rw-r--r-- 1 200096 wheel    0 Feb  5 17:03 ut
[tdh@mrx ipk]> sudo umount /mnt/mrbill/NFS4
[tdh@mrx ipk]> sudo mount -o vers=4 mrbill:/home/NFS4 /mnt/mrbill/NFS4
'vers=4' is not supported.  Use '-t nfs4' instead.
[tdh@mrx ipk]> sudo mount -t nfs4 mrbill:/home/NFS4 /mnt/mrbill/NFS4
mount.nfs4: mount point /mnt/mrbill/NFS4 does not exist

Okay, mrbill knows nothing about NFSv4 as far as I can tell:

root@mrbill:/home/NFS4# mount -t nfs4 sandman:/export/home /mnt/sandman/home
mount: unknown filesystem type 'nfs4'

I'm sensing protocol discrimination here:

root@mrbill:/home/NFS4# ipkg list | grep -i nfs
kernel-module-lockd - 2.6.16-r6.6 - lockd kernel module; NFS file locking service version 0.5.
kernel-module-nfs - 2.6.16-r6.6 - nfs kernel module
kernel-module-nfs - 2.6.16-r6.4 -
kernel-module-nfsd - 2.6.16-r6.6 - nfsd kernel module
nfs-utils - 1.0.6-r7 - userspace utilities for kernel nfs
nfs-utils-doc - 1.0.6-r7 - userspace utilities for kernel nfs

Time to check the log file:

Feb 12 09:08:29 (none) user.warn kernel: nfsd: nfsv4 idmapping failing: has idmapd not been started?

Okay, configure idmapping and reboot:

Feb 12 09:16:37 (none) user.info kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Feb 12 09:16:37 (none) user.warn kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Feb 12 09:16:37 (none) user.warn kernel: NFSD: unable to find recovery directory /var/lib/nfs/v4recovery
Feb 12 09:16:37 (none) user.warn kernel: NFSD: starting 90-second grace period

Try the mount again:

[tdh@mrx ipk]> sudo mount -t nfs4 mrbill:/home/NFS4 /mnt/mrbill/NFS4
mount.nfs4: Permission denied

And try it from a Solaris client:

[tdh@sandman keytabs]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4
[tdh@sandman keytabs]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
nfs mount: mount: /mnt/mrbill/NFS4: Permission denied

Okay, can we get Kerberos working at all on the NSLU2?

root@mrbill:~# more /etc/exports
/home/NFS4 192.168.2.0/24(rw,fsid=0,sec=krb5,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
root@mrbill:~# exportfs -rv
exportfs: /etc/exports:1: unknown keyword "sec=krb5"
unexporting sandman.internal.excfb.com:/home/NFS4 from kernel

The keyword is not correct? Time to try on a known good linux config:

[tdh@mrx ipk]> cat /etc/exports
/home/tdh 192.168.2.0/24(rw,fsid=0,sec=krb5,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
[tdh@mrx ipk]> sudo exportfs -rv
exportfs: /etc/exports:1: unknown keyword "sec=krb5"

Okay, here is what we are supposed to do:

[tdh@mrx ipk]> cat /etc/exports
/home/tdh gss/krb5(rw,fsid=0,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
[tdh@mrx ipk]> sudo exportfs -rv
exporting gss/krb5:/home/tdh
exporting gss/krb5:/home/tdh to kernel
gss/krb5:/home/tdh: Cannot allocate memory

By sheer effort of will, I determined that the firewall was on.

root@mrbill:~# showmount -e mrx
Export list for mrx:
/home/tdh gss/krb5

First lets see what happens without kerberos:

[tdh@sandman ~]> sudo mount -o vers=3 mrx:/home/tdh /mnt/mrx/tdh
[tdh@sandman ~]> ls -la /mnt/mrx/tdh
total 230394
drwxr-xr-x   7 tdh      staff       4096 Feb 12 02:01 .
drwxr-xr-x   3 root     root         512 Feb 12 11:49 ..

And NFSv4:

[tdh@sandman ~]> sudo mount mrx:/home/tdh /mnt/mrx/tdh
nfs mount: mrx:/home/tdh: No such file or directory

Okay, I knew about this, but forgot it. I think I heard Bruce complaining about still having it:

[tdh@sandman ~]> sudo mount mrx:/ /mnt/mrx/tdh
[tdh@sandman ~]> ls -al /mnt/mrx/tdh
total 230394
drwxr-xr-x   7 tdh      nobody      4096 Feb 12 02:01 .
drwxr-xr-x   3 root     root         512 Feb 12 11:49 ..
-rw-------   1 tdh      nobody        68 Feb 12 01:51 .Xauthority
-rw-------   1 tdh      nobody        96 Feb 12 11:31 .lesshst

And now we turn on kerberos:

[tdh@sandman ~]> sudo mount mrx:/ /mnt/mrx/tdh
NFS compound failed for server mrx: error 7 (RPC: Authentication error)
NFS compound failed for server mrx: error 7 (RPC: Authentication error)
NFS compound failed for server mrx: error 7 (RPC: Authentication error)
nfs mount: mount: /mnt/mrx/tdh: Permission denied

We can be very specific about what security flavor we want to use:

[tdh@sandman ~]> sudo mount -o sec=krb5 mrx:/ /mnt/mrx/tdh
nfs mount: mount: /mnt/mrx/tdh: Permission denied

Note that the compound fails messages must have been about AUTH_NONE, AUTH_SYS, and AUTH_DH.

I think I've found the answer in Mike Eisler's blog Real Authentication in NFS, scroll down into the comments:

> Also, does NetApp require a root principle like Solaris did prior to 10?

Actually even prior to Solaris 10, the Solaris NFS server would allow
an NFSv3 mount if root didn't have Kerberos credentials. ONTAP is the
same way. However, if using NFSv4, because NFSv4 has no separate mount
protocol, an NFSv4 server cannot distinguish a mount from a LOOKUP. If
a volume is exported with sec=krb5, then the NFSv4 requests need to be
using Kerberos. Since UNIX clients usually require one to be superuser
to do an NFS mount, superuser (root) needs to have credentials. Root
credentials aren't required, but whatever uid the credentials map to
has to have search permissions for the path name.

And we can try that here:

kadmin:  addprinc root
WARNING: no policy specified for root@INTERNAL.EXCFB.COM; defaulting to no policy
Enter password for principal "root@INTERNAL.EXCFB.COM":
Re-enter password for principal "root@INTERNAL.EXCFB.COM":
Principal "root@INTERNAL.EXCFB.COM" created.

And then we grab a ticket:

[tdh@sandman ~]> sudo kinit root
Password for root@INTERNAL.EXCFB.COM:
[tdh@sandman ~]> sudo mount -o sec=krb5 mrx:/ /mnt/mrx/tdh

Aargh!

[tdh@sandman ~]> ls -la /mnt/mrx/tdh
total 230394
drwxr-xr-x   7 tdh      nobody      4096 Feb 12 02:01 .
drwxr-xr-x   3 root     root         512 Feb 12 11:49 ..
-rw-------   1 tdh      nobody        68 Feb 12 01:51 .Xauthority
-rw-------   1 tdh      nobody        96 Feb 12 11:31 .lesshst

Since we can't even get the export shared without kerberos on mrbill, that does not explain the issue on that machine.

This works:

[tdh@sandman ~]> sudo mount -o vers=3 mrbill:/home/NFS4 /mnt/mrbill/NFS4

And this does not:

[tdh@sandman ~]> sudo mount -o vers=4 mrbill:/ /mnt/mrbill/NFS4
nfs mount: mount: /mnt/mrbill/NFS4: Resource temporarily unavailable

I'll come back to this later...


Originally posted on Kool Aid Served Daily
Copyright (C) 2007, Kool Aid Served Daily

Sunday Feb 11, 2007

Installing a Kerberos KDC and setting up NFS mounts

We always seem to have problems at Connectathon setting up Kerberos. So I decided to take the cookbook we use there and get kerberos working on my home systems. Please note that I could easily clean up the notes to not show some errors I make. But then, where is the love?

Also, as with any first foray into a new tool, I have no clue what I am doing. I kinda understand tickets and the ideas behind Kerberos, but I'm really in the dark as to what I'm supposed to do.

First edit /etc/krb5/krb5.conf:

# diff krb5.conf stock/krb5.conf
35c35
<         default_realm = INTERNAL.EXCFB.COM
---
>         default_realm = ___default_realm___
38,41c38,43
<         INTERNAL.EXCFB.COM = {
<                 kdc = sandman.internal.excfb.com
<                 kdc = ultralord.internal.excfb.com
<                 admin_server = sandman.internal.excfb.com
---
>         ___default_realm___ = {
>                 kdc = ___master_kdc___
>                 kdc = ___slave_kdc1___
>                 kdc = ___slave_kdc2___
>                 kdc = ___slave_kdcN___
>                 admin_server = ___master_kdc___

Then edit /etc/krb5/kdc.conf:

# diff kdc.conf stock/kdc.conf
32c32
<       INTERNAL.EXCFB.COM = {
---
>       ___default_realm___ = {
41,42d40
<               sunw_dbprob_enable = true
<               sunw_dbprop_master_ulogsize = 1000

Make sure you can get at the kdcs via DNS (or whatever name service in /etc/resolv.conf)

# host sandman
sandman.internal.excfb.com has address 192.168.2.109
# host sandman.internal.excfb.com
sandman.internal.excfb.com has address 192.168.2.109

Create the kerberos database

# /usr/sbin/kdb5_util create -r INTERNAL.EXCFB.COM -s
Initializing database '/var/krb5/principal' for realm 'INTERNAL.EXCFB.COM',
master key name 'K/M@INTERNAL.EXCFB.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

Start getting some principals:

# /usr/sbin/kadmin.local
Authenticating as principal root/admin@INTERNAL.EXCFB.COM with password.
kadmin.local:  addprinc tdh/admin
WARNING: no policy specified for tdh/admin@INTERNAL.EXCFB.COM; defaulting to no policy
Enter password for principal "tdh/admin@INTERNAL.EXCFB.COM":
Re-enter password for principal "tdh/admin@INTERNAL.EXCFB.COM":
Principal "tdh/admin@INTERNAL.EXCFB.COM" created.

Get some kiprop installed:

kadmin.local:  addprinc -randkey kiprop/sandman.internal.excfb.com
WARNING: no policy specified for kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM".
kadmin.local:  addprinc -randkey kiprop/ultralord.internal.excfb.com
WARNING: no policy specified for kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM" created.

Enable kadmin and changepw:

kadmin.local:  ktadd -k /etc/krb5/kadm.keytab kadmin/sandman.internal.excfb.com
Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
kadmin.local:  ktadd -k /etc/krb5/kadm.keytab changepw/sandman.internal.excfb.com
Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab.

Enable kiprop:

kadmin.local:  ktadd -k /etc/krb5/kadm.keytab kiprop/sandman.internal.excfb.com
Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab.

Quit:

kadmin.local:  quit

Enable the services:

# svcadm enable -r network/security/krb5kdc
# svcadm enable -r network/security/kadmin

Authenticate the admin account:

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin: Communication failure with server while initializing kadmin interface

Hmm, I got the right password. I can see what happens when it is wrong:

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin: Incorrect password while initializing kadmin interface

Ahh, lets see if kerberos is up and running:

# grep kadmin /var/adm/messages
Feb 11 23:31:19 sandman svc.startd[7]: [ID 748625 daemon.error] network/security/kadmin:default failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details)
Feb 11 23:31:57 sandman kadmin[4143]: [ID 737709 user.error] unable to open connection to ADMIN server (t_error 9)
Feb 11 23:33:56 sandman kadmin[4146]: [ID 737709 user.error] unable to open connection to ADMIN server (t_error 9)

No, it is not.

# svcs -xv
svc:/network/security/kadmin:default (Kerberos administration daemon)
 State: maintenance since Sun Feb 11 23:31:19 2007
Reason: Restarting too quickly.
   See: http://sun.com/msg/SMF-8000-L5
   See: man -M /usr/share/man -s 1M kadmind
   See: /var/svc/log/network-security-kadmin:default.log
Impact: This service is not running.

Clear the maintenance state:

# svcadm clear /network/security/kadmin:default

Restart:

# svcadm enable -r network/security/kadmin

Check:

# svcs -xv #

And try again:

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin: Communication failure with server while initializing kadmin interface

If we look at kadm5.acl:

\*/admin@___default_realm___ \*

Hmm, touch that up:

\*/admin@INTERNAL.EXCFB.COM \*

And for sanity:

# grep default \*
kdc.conf:[kdcdefaults]
kdc.conf:               default_principal_flags = +preauth
krb5.conf:[libdefaults]
krb5.conf:        default_realm = INTERNAL.EXCFB.COM
krb5.conf:      ___domainname___ = ___default_realm___
krb5.conf:        default = FILE:/var/krb5/kdc.log
krb5.conf:[appdefaults]

Okay, time to fix up krb5.conf as well:

[domain_realm]
        ___domainname___ = INTERNAL.EXCFB.COM

And restart:

# svcadm restart network/security/krb5kdc
# svcadm restart network/security/kadmin

And try again:

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin: Communication failure with server while initializing kadmin interface

Okay, we know it is talking to something, i.e., it understands a bad password.

Lets try something else:

# kadmin.local
Authenticating as principal root/admin@INTERNAL.EXCFB.COM with password.
kadmin.local:  addprinc admin/admin@INTERNAL.EXCFB.COM
WARNING: no policy specified for admin/admin@INTERNAL.EXCFB.COM; defaulting to no policy
Enter password for principal "admin/admin@INTERNAL.EXCFB.COM":
Re-enter password for principal "admin/admin@INTERNAL.EXCFB.COM":
Principal "admin/admin@INTERNAL.EXCFB.COM" created.
kadmin.local:  quit

Okay, time to search. If we look at System Administration Guide: Security Services :

Communication failure with server while initializing kadmin interface

    Cause: The host that was entered for the admin server, also called the master KDC,
    did not have the kadmind daemon running.

    Solution: Make sure that you specified the correct host name for the master KDC.
    If you specified the correct host name, make sure that kadmind is running on
    the master KDC that you specified.

But wait:

# svcs | grep krb
online         23:43:04 svc:/network/security/krb5kdc:default
# svcs | grep kad
maintenance    23:42:54 svc:/network/security/kadmin:default
# svcs -vx
svc:/network/security/kadmin:default (Kerberos administration daemon)
 State: maintenance since Sun Feb 11 23:42:54 2007
Reason: Restarting too quickly.
   See: http://sun.com/msg/SMF-8000-L5
   See: man -M /usr/share/man -s 1M kadmind
   See: /var/svc/log/network-security-kadmin:default.log
Impact: This service is not running.

Lets look at the log file:

Feb 11 23:42:53 sandman kadmind[4275](Error): Keytab file "/etc/krb5/kadm5.keytab" does not exist
Feb 11 23:42:53 sandman kadmind[4275](Error): Keytab file "/etc/krb5/kadm5.keytab" does not exist
Feb 11 23:42:53 sandman kadmind[4275](info): No dictionary file specified, continuing without one.
Feb 11 23:42:53 sandman kadmind[4275](Error): Unable to set RPCSEC_GSS service names ('kadmin@sandman.internal.excfb.com,changepw@sandman.internal.excfb.com')
krb5kdc: Interrupted system call - while selecting for network input(1)
Feb 11 23:43:03 sandman krb5kdc[4105](info): shutting down

Hmm, we need to create a keytab:

# ls -la /etc/krb5/kadm5.keytab
/etc/krb5/kadm5.keytab: No such file or directory

Ack, why do I have a kadm.keytab and not a kadm5.keytab?

# mv kadm.keytab kadm5.keytab

Because that is what I frigging entered in my session!

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin:

The correct incantations should have been:

kadmin.local:  ktadd -k /etc/krb5/kadm5.keytab kadmin/sandman.internal.excfb.com
kadmin.local:  ktadd -k /etc/krb5/kadm5.keytab changepw/sandman.internal.excfb.com
kadmin.local:  ktadd -k /etc/krb5/kadm5.keytab kiprop/sandman.internal.excfb.com

Okay, back to our regularly scheduled programming:

What principals exist?

kadmin:  listprincs
K/M@INTERNAL.EXCFB.COM
admin/admin@INTERNAL.EXCFB.COM
changepw/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
kadmin/changepw@INTERNAL.EXCFB.COM
kadmin/history@INTERNAL.EXCFB.COM
kadmin/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
krbtgt/INTERNAL.EXCFB.COM@INTERNAL.EXCFB.COM
tdh/admin@INTERNAL.EXCFB.COM

To kerberize NFS, we need to touch up /etc/nfssec.conf:

# diff nfssec.conf nfssec.conf.stock
48,50c48,50
< krb5          390003  kerberos_v5     default -               # RPCSEC_GSS
< krb5i         390004  kerberos_v5     default integrity       # RPCSEC_GSS
< krb5p         390005  kerberos_v5     default privacy         # RPCSEC_GSS
---
> #krb5         390003  kerberos_v5     default -               # RPCSEC_GSS
> #krb5i                390004  kerberos_v5     default integrity       # RPCSEC_GSS
> #krb5p                390005  kerberos_v5     default privacy         # RPCSEC_GSS

We need to add a nfs principal:

kadmin:  addprinc -randkey nfs/sandman.internal.excfb.com
WARNING: no policy specified for nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM" created.
kadmin:  ktadd nfs/sandman.internal.excfb.com
Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/krb5.keytab.
Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/krb5.keytab.
Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/krb5.keytab.
Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/krb5.keytab.

Verify that is does indeed exist:

# klist -k
Keytab name: FILE:/etc/krb5/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
   3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
   3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
   3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM

And now we are going to have to make a share that is kerberized and setup a client to access it:

# /usr/sbin/kclient

Starting client setup

---------------------------------------------------
Do you want to use DNS for kerberos lookups ? [y/n]: n
        No action performed.
Enter the Kerberos realm: INTERNAL.EXCFB.COM
Specify the KDC hostname for the above realm: sandman.internal.excfb.com
sandman.internal.excfb.com

Note, this system and the KDC's time must be within 5 minutes of each other for Kerberos to function.  Both systems should run some form of time
 synchronization system like Network Time Protocol (NTP).

Setting up /etc/krb5/krb5.conf.

Enter the krb5 administrative principal to be used: tdh/admin
Obtaining TGT for tdh/admin ...
Password for tdh/admin@INTERNAL.EXCFB.COM:

Do you have multiple DNS domains spanning the Kerberos realm INTERNAL.EXCFB.COM ? [y/n]: n
        No action performed.

Do you plan on doing Kerberized nfs ? [y/n]: y

nfs/ultralord.internal.excfb.com entry ADDED to KDC database.
nfs/ultralord.internal.excfb.com entry ADDED to keytab.

host/ultralord.internal.excfb.com entry ADDED to KDC database.
host/ultralord.internal.excfb.com entry ADDED to keytab.

Do you want to copy over the master krb5.conf file ? [y/n]: y
Enter the pathname of the file to be copied: /etc/krb5/krb5.conf
cp: /etc/krb5/krb5.conf and /etc/krb5/krb5.conf are identical

Copy of /etc/krb5/krb5.conf failed, exiting.
---------------------------------------------------
Setup FAILED.

Hmm, how are we supposed to enter that? I bet we need to use /net. Which I don't have configured right now. Okay, the hard way:

# scp sandman:/etc/krb5/krb5.conf /etc/krb5/krb5.conf

Now, lets set up a test share:

# cd /export
# mkdir kerberos
# cd kerberos
# touch see_me
# chown tdh:staff see_me
# ls -la
total 4
drwxr-xr-x   2 root     root         512 Feb 12 00:23 .
drwxr-xr-x   4 root     sys          512 Feb 12 00:23 ..
-rw-r--r--   1 tdh      staff          0 Feb 12 00:23 see_me
# share -F nfs -o sec=krb5:krb5i:krb5p -d "Kerberos" /export/kerberos
# share -F nfs -d "Home dirs" /export/home
# share
-               /export/kerberos   sec=krb5,sec=krb5i,sec=krb5p   "Kerberos"
-               /export/home   rw   "Home dirs"

Now try to get some access:

[tdh@ultralord ~]> kinit
kinit(v5): Client not found in Kerberos database while getting initial credentials
[tdh@ultralord ~]> sudo klist -k
Keytab name: FILE:/etc/krb5/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM

Okay, I think I need to add user principals for tdh:

kadmin:  addprinc tdh
WARNING: no policy specified for tdh@INTERNAL.EXCFB.COM; defaulting to no policy
Enter password for principal "tdh@INTERNAL.EXCFB.COM":
Re-enter password for principal "tdh@INTERNAL.EXCFB.COM":
Principal "tdh@INTERNAL.EXCFB.COM" created.

[tdh@ultralord ~]> kinit
Password for tdh@INTERNAL.EXCFB.COM:

And now I want to get a mount:

[tdh@ultralord ~]> sudo mkdir -p /mnt/sandman/home
[tdh@ultralord ~]> sudo mkdir -p /mnt/sandman/kerberos
[tdh@ultralord ~]> sudo showmount -e sandman
export list for sandman:
/export/kerberos (everyone)
/export/home     (everyone)
[tdh@ultralord ~]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos
[tdh@ultralord ~]> sudo mount sandman:/export/home /mnt/sandman/home
[tdh@ultralord ~]> ls -al /mnt/sandman/kerberos
total 4
drwxr-xr-x   2 root     root         512 Feb 12 00:23 .
drwxr-xr-x   4 root     root         512 Feb 12 00:36 ..
-rw-r--r--   1 tdh      staff          0 Feb 12 00:23 see_me
[tdh@ultralord ~]> ls -la /mnt/sandman/home
total 22
drwxr-xr-x   4 root     root         512 Dec 30 15:01 .
drwxr-xr-x   4 root     root         512 Feb 12 00:36 ..
drwx------   2 root     root        8192 Dec 20 11:28 lost+found
drwxr-xr-x   4 tdh      staff        512 Jan 21 20:48 tdh

Success!

But wait, we need to show that a client without kerberos enabled will be denied access to sandman:/export/kerberos:

[tdh@kanigix ~]> sudo mkdir -p /mnt/sandman/home
[tdh@kanigix ~]> sudo mkdir -p /mnt/sandman/kerberos
[tdh@kanigix ~]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos
nfs mount: mount: /mnt/sandman/kerberos: Permission denied

Some other things to do would be to setup /etc/pam.conf to allow single signon - i.e., use ssh without a password. We also need to setup ultralord as a slave.

But before I tune this out, we need to get a Linux client up and running. Why? Because we need to show we can interoperate.

Some systems only support single DES, so we need to create special keytabs for them:

kadmin:  addprinc -randkey nfs/mrx.internal.excfb.com
WARNING: no policy specified for nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM" created.
kadmin:  addprinc -randkey host/mrx.internal.excfb.com
WARNING: no policy specified for host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM" created.

Now, I've created /export/keytabs to store the keytab files we will need:

# cd /export
# mkdir keytabs
# share -F nfs -o ro /export/keytabs

And we can create the keytab:

kadmin:  ktadd -k /export/keytabs/mrx.keytab -e des-cbc-crc:normal nfs/mrx.internal.excfb.com
Entry for principal nfs/mrx.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrx.keytab.
kadmin:  ktadd -k /export/keytabs/mrx.keytab -e des-cbc-crc:normal host/mrx.internal.excfb.com
Entry for principal host/mrx.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrx.keytab.

We see we are in business:

# cp /etc/krb5/krb5.conf /export/keytabs/
# ls -la
total 10
drwxr-xr-x   2 root     root         512 Feb 12 00:50 .
drwxr-xr-x   5 root     sys          512 Feb 12 00:46 ..
-rw-r--r--   1 root     root        1968 Feb 12 00:50 krb5.conf
-rw-------   1 root     root         155 Feb 12 00:48 mrx.keytab
# chmod +r mrx.keytab

And now we setup the Linux machine:

[root@mrx ~]# mkdir -p /mnt/sandman/keytabs
[root@mrx ~]# showmount -e sandman
Export list for sandman:
/export/kerberos (everyone)
/export/home     (everyone)
/export/keytabs  (everyone)
[root@mrx ~]# mount sandman:/export/keytabs /mnt/sandman/keytabs

We should make sure we do not have access to sandman:/export/kerberos:

[root@mrx ~]# mkdir -p /mnt/sandman/kerberos
[root@mrx ~]# mkdir -p /mnt/sandman/home
[root@mrx ~]# mount sandman:/export/kerberos /mnt/sandman/kerberos
mount: sandman:/export/kerberos failed, security flavor not supported

What do we need to change:

[root@mrx ~]# cd /etc
[root@mrx etc]# ls -la k\*
-rw-r--r-- 1 root root  657 Jan  9 14:03 krb5.conf
-rw-r--r-- 1 root root 2241 Jul 13  2006 krb.conf
-rw-r--r-- 1 root root 1296 Jul 13  2006 krb.realms
[root@mrx etc]# mkdir stock
[root@mrx etc]# cp k\* stock
[root@mrx etc]# cp /mnt/sandman/keytabs/krb5.conf .
cp: overwrite `./krb5.conf'? y
[root@mrx etc]# cp /mnt/sandman/keytabs/mrx.keytab krb5.keytab

And we try to authenticate:

[tdh@mrx ~]> kinit
kinit: Command not found.

Okay, we need to install the kerberos packages:

[tdh@mrx /]> sudo yum install krb5-workstation
Loading "installonlyn" plugin
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Nothing to do

No, we don't. Where is that rascally rabbit?

[tdh@mrx /]> sudo find . -name kinit
./usr/kerberos/bin/kinit
[tdh@mrx /]> ./usr/kerberos/bin/kinit
Password for tdh@INTERNAL.EXCFB.COM:

And we try the mount:

[tdh@mrx /]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos
mount: sandman:/export/kerberos failed, security flavor not supported
[tdh@mrx /]> ./usr/kerberos/bin/klist
Ticket cache: FILE:/tmp/krb5cc_1066
Default principal: tdh@INTERNAL.EXCFB.COM

Valid starting     Expires            Service principal
02/12/07 01:01:42  02/12/07 09:01:42  krbtgt/INTERNAL.EXCFB.COM@INTERNAL.EXCFB.COM
        renew until 02/13/07 00:59:17



Kerberos 4 ticket cache: /tmp/tkt1066
klist: You have no tickets cached

What is up here?

# snoop -x 0,2000 -o /tmp/m2s.snoop sandman mrx
Using device /dev/hme (promiscuous mode)
33 \^C

Note: I used -x 0,2000 to get payload data. I knew I would want to look at most of the packet.

And

[tdh@mrx ~]> sudo mount -t nfs4 sandman:/export/kerberos /mnt/sandman/kerberos
mount.nfs4: Operation not permitted

 26   0.00034 mrx.internal.excfb.com -> sandman      NFS C 4 () PUTFH FH=324D LOOKUP export GETFH GETATTR 10011a 30a23a
 27   0.00030      sandman -> mrx.internal.excfb.com NFS R 4 () NFS4_OK PUTFH NFS4_OK LOOKUP NFS4_OK GETFH NFS4_OK FH=30E6 GETATTR NFS4_OK
 28   0.00033 mrx.internal.excfb.com -> sandman      NFS C 4 () PUTFH FH=30E6 LOOKUP kerberos GETFH GETATTR 10011a 30a23a
 29   0.00021      sandman -> mrx.internal.excfb.com NFS R 4 () NFS4ERR_WRONGSEC PUTFH NFS4_OK LOOKUP NFS4ERR_WRONGSEC

I popped into wireshark and I found out that mrx is only sending AUTH_SYS and AUTH_NULL.

Note: I used wireshark because it will parse the payload data for me. I didn't want to be doing byte conversions and consulting some specs!

In NetApp Filer, NFSv4, and Linux, we find using -o sec=krb5. We can try that:

[tdh@mrx ~]> sudo mount -t nfs4 -o sec=krb5 sandman:/export/kerberos /mnt/sandman/kerberos
Warning: rpc.gssd appears not to be running.
mount.nfs4: Invalid argument

Which is strange, since it is running:

[tdh@mrx ~]> sudo chkconfig --list | grep rpcgssd
rpcgssd         0:off   1:off   2:off   3:on    4:on    5:on    6:off
[tdh@mrx ~]> sudo chkconfig --list | grep rpcidmapd
rpcidmapd       0:off   1:off   2:off   3:on    4:on    5:on    6:off

What does the log state:

RPC: Couldn't create auth handle (flavor 390003)

I've copied the stock krb5.conf back and now the diffs are:

[tdh@mrx /etc]> diff krb5.conf stock/krb5.conf
7c7
<  default_realm = INTERNAL.EXCFB.COM
---
>  default_realm = EXAMPLE.COM
14,17c14,17
<  INTERNAL.EXCFB.COM = {
<   kdc = sandman.internal.excfb.com:88
<   admin_server = sandman.internal.excfb.com:749
<   default_domain = internal.excfb.com
---
>  EXAMPLE.COM = {
>   kdc = kerberos.example.com:88
>   admin_server = kerberos.example.com:749
>   default_domain = example.com
21,22c21,22
<  .internal.excfb.com = INTERNAL.EXCFB.COM
<  internal.excfb.com = INTERNAL.EXCFB.COM
---
>  .example.com = EXAMPLE.COM
>  example.com = EXAMPLE.COM

You know what, rpc.gssd is not running!

[tdh@mrx /etc]> ps -ef | grep rpc
rpc       1877     1  0 01:49 ?        00:00:00 portmap
root      1898     1  0 01:49 ?        00:00:00 rpc.statd
root      1931     1  0 01:49 ?        00:00:00 rpc.idmapd
tdh       2697  2519  0 02:04 pts/0    00:00:00 grep rpc

[tdh@mrx /etc]> sudo sh -c "ulimit -c unlimited;/usr/sbin/rpc.gssd -f -vvv"
Using keytab file '/etc/krb5.keytab'
Processing keytab entry for principal 'nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM'
We will use this entry (nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM)
Processing keytab entry for principal 'host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM'
We will NOT use this entry (host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM)
Using (machine) credentials cache: 'MEMORY:/tmp/krb5cc_machine_INTERNAL.EXCFB.COM'

And I put it in the background. Hmm, why doesn't it like the host entry?

Alright, I went back to why isn't rpc.gssd starting up at boot:

[ -f /etc/sysconfig/nfs ] && . /etc/sysconfig/nfs
[ "${SECURE_NFS}" != "yes" ] && exit 0

# ls -la /etc/sysconfig/nfs
#

Time to create it (look at Learning NFSv4 with Fedora Core 2 (Linux 2.6. 5 kernel))

# This entry should be "yes" if you are using RPCSEC_GSS_KRB5 (auth=krb5,krb5i, or krb5p)
SECURE_NFS="yes"
# This entry sets the number of NFS server processes.  8 is the default
RPCNFSDCOUNT=8

[tdh@mrx sysconfig]> sudo /etc/init.d/rpcgssd start
Starting RPC gssd:                                         [  OK  ]

God I'm totally hacked about this:

[tdh@mrx sysconfig]> sudo mount -o sec=krb5 sandman:/export/kerberos /mnt/sandman/kerberos
[tdh@mrx sysconfig]> ls -la /mnt/sandman/kerberos
total 5
drwxr-xr-x 2 root root   512 Feb 12 00:23 .
drwxr-xr-x 5 root root  4096 Feb 12 00:49 ..
-rw-r--r-- 1 tdh  wheel    0 Feb 12 00:23 see_me

Originally posted on Kool Aid Served Daily
Copyright (C) 2007, Kool Aid Served Daily

Sunday Dec 31, 2006

How NFSv4 should work when crossing filesystems

In Some fun with NFSv4 and automount across a ssh tunnel, I revealed the work going on in Solaris for Mirror Mounts. The example was a desire to automount across a ssh tunnel. Well, I dusted off wont, the box from hell (being used by my son for video games) and created some zfs filesystems on it:

# zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
zoo                398K   118G  24.5K  /zoo
zoo/home           256K   118G  35.5K  /export/zfs
zoo/home/braves   24.5K   118G  24.5K  /export/zfs/braves
zoo/home/kanigix  24.5K   118G  24.5K  /export/zfs/kanigix
zoo/home/loghyr   24.5K   118G  24.5K  /export/zfs/loghyr
zoo/home/mrx      24.5K   118G  24.5K  /export/zfs/mrx
zoo/home/nfsv2    24.5K   118G  24.5K  /export/zfs/nfsv2
zoo/home/nfsv3    24.5K   118G  24.5K  /export/zfs/nfsv3
zoo/home/nfsv4    24.5K   118G  24.5K  /export/zfs/nfsv4
zoo/home/spud     24.5K   118G  24.5K  /export/zfs/spud
zoo/home/tdh      24.5K   118G  24.5K  /export/zfs/tdh
# uname -a
SunOS wont 5.11 snv_55 i86pc i386 i86pc

I then opened a ssh tunnel to it on my Fedora Core 4 box and did a little bit of exploring:

[tdh@adept tdh]> uname -a
Linux adept 2.6.15-1.1833_FC4 #1 Wed Mar 1 23:41:37 EST 2006 i686 i686 i386 GNU/Linux
[tdh@adept ~/usenix]> ssh -fN -L "5049:wont:2049" wont
Password:
[tdh@adept ~/usenix]> sudo mount -o port=5049 -t nfs4 localhost:/ /nfs4/wont
[tdh@adept ~/usenix]> cd /nfs4/wont
[tdh@adept wont]> ls -la
total 6
drwxr-xr-x  38 root root 1024 Dec 31 17:49 .
drwxr-xr-x   4 root root 4096 Dec 31 18:17 ..
drwxr-xr-x   4 root sys   512 Dec 31 17:50 export
[tdh@adept wont]> cd export
[tdh@adept export]> ls -la
total 4
drwxr-xr-x   4 root sys   512 Dec 31 17:50 .
drwxr-xr-x  38 root root 1024 Dec 31 17:49 ..
drwxr-xr-x  11 root sys    11 Dec 31 17:50 zfs
[tdh@adept export]> cd zfs
[tdh@adept zfs]> ls -la
total 16
drwxr-xr-x  11 root sys  11 Dec 31 17:50 .
drwxr-xr-x   4 root sys 512 Dec 31 17:50 ..
drwxr-xr-x   2 root sys   2 Dec 31 17:50 braves
drwxr-xr-x   2 root sys   2 Dec 31 17:50 kanigix
drwxr-xr-x   2 root sys   2 Dec 31 17:50 loghyr
drwxr-xr-x   2 root sys   2 Dec 31 17:50 mrx
drwxr-xr-x   2 root sys   2 Dec 31 17:50 nfsv2
drwxr-xr-x   2 root sys   2 Dec 31 17:50 nfsv3
drwxr-xr-x   2 root sys   2 Dec 31 17:50 nfsv4
drwxr-xr-x   2 root sys   2 Dec 31 17:50 spud
drwxr-xr-x   2 root sys   2 Dec 31 17:50 tdh
[tdh@adept zfs]> cd tdh
[tdh@adept tdh]> ls -la
total 3
drwxr-xr-x   2 root sys  2 Dec 31 17:50 .
drwxr-xr-x  11 root sys 11 Dec 31 17:50 ..

Notice that I only did one mount command. As I crossed down into the exported filesystems, the Linux 2.16 implementation of NFSv4 did the mounts automatically for me in the background. Also, note that since '/' is not exported from wont, this must be a pseudo-fs:

[tdh@adept tdh]> showmount -e wont
Export list for wont:
/export/zfs         (everyone)
/export/zfs/nfsv2   (everyone)
/export/zfs/nfsv3   (everyone)
/export/zfs/nfsv4   (everyone)
/export/zfs/tdh     (everyone)
/export/zfs/loghyr  (everyone)
/export/zfs/kanigix (everyone)
/export/zfs/mrx     (everyone)
/export/zfs/spud    (everyone)
/export/zfs/braves  (everyone)

Let's export '/' and see what happens:

# share -F nfs -o rw -d "root" /

And on the Linux box:

[tdh@adept tdh]> cd /nfs4/wont
[tdh@adept wont]> ls -la
total 6
drwxr-xr-x  38 root root 1024 Dec 31 17:49 .
drwxr-xr-x   4 root root 4096 Dec 31 18:17 ..
drwxr-xr-x   4 root sys   512 Dec 31 17:50 export

What happened? Why didn't we see the root directory on wont? Well, when we did the mount command earlier, we basically got a reference to a file handle in the pseudo-fs. We need to flush this by umounting and remounting:

[tdh@adept wont]> cd
[tdh@adept ~]> sudo umount /nfs4/wont/
[tdh@adept ~]> sudo mount -o port=5049 -t nfs4 localhost:/ /nfs4/wont
[tdh@adept ~]> cd /nfs4/wont
[tdh@adept wont]> ls -la
total 67
drwxr-xr-x  38 root root 1024 Dec 31 17:49 .
drwxr-xr-x   4 root root 4096 Dec 31 18:17 ..
lrwxrwxrwx   1 root root    9 Dec 31 13:17 bin -> ./usr/bin
drwxr-xr-x   5 root sys   512 Dec 31 14:12 boot
drwxr-xr-x   2 root root  512 Dec 31 14:51 Desktop
drwxr-xr-x  24 root sys  4096 Dec 31 14:42 dev
drwxr-xr-x  10 root sys   512 Dec 31 14:42 devices
drwxr-xr-x   2 root root  512 Dec 31 14:51 Documents
drwxr-xr-x   9 root root  512 Dec 31 17:31 .dt
-rwxr-xr-x   1 root root 5111 Dec 31 14:51 .dtprofile
-rw-------   1 root root   16 Dec 31 17:31 .esd_auth
drwxr-xr-x  87 root sys  4608 Dec 31 17:52 etc
drwxr-xr-x   4 root sys   512 Dec 31 17:50 export
...

Let's walk down the paths again and see what happens:

[tdh@adept wont]> cd export [tdh@adept export]> ls -la total 5 drwxr-xr-x 4 root sys 512 Dec 31 17:50 . drwxr-xr-x 38 root root 1024 Dec 31 17:49 .. drwxr-xr-x 2 root root 512 Dec 31 13:17 home drwxr-xr-x 11 root sys 11 Dec 31 17:50 zfs [tdh@adept export]> cd zfs [tdh@adept zfs]> ls -la total 16 drwxr-xr-x 11 root sys 11 Dec 31 17:50 . drwxr-xr-x 4 root sys 512 Dec 31 17:50 .. drwxr-xr-x 2 root sys 2 Dec 31 17:50 braves drwxr-xr-x 2 root sys 2 Dec 31 17:50 kanigix drwxr-xr-x 2 root sys 2 Dec 31 17:50 loghyr drwxr-xr-x 2 root sys 2 Dec 31 17:50 mrx drwxr-xr-x 2 root sys 2 Dec 31 17:50 nfsv2 drwxr-xr-x 2 root sys 2 Dec 31 17:50 nfsv3 drwxr-xr-x 2 root sys 2 Dec 31 17:50 nfsv4 drwxr-xr-x 2 root sys 2 Dec 31 17:50 spud drwxr-xr-x 2 root sys 2 Dec 31 17:50 tdh [tdh@adept zfs]> cd tdh [tdh@adept tdh]> ls -la total 3 drwxr-xr-x 2 root sys 2 Dec 31 17:50 . drwxr-xr-x 11 root sys 11 Dec 31 17:50 ..

Let's make sure we are in the right place:

# scp sandman:/export/home/tdh/.tcshrc .
Password:
.tcshrc              100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  5417       00:00
# chown tdh:staff .tcshrc
# ls -la
total 18
drwxr-xr-x   2 root     sys            3 Dec 31 18:10 .
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 ..
-rw-------   1 tdh      staff       5417 Dec 31 18:10 .tcshrc

And on the client:

[tdh@adept tdh]> ls -la
total 9
drwxr-xr-x   2 root sys       3 Dec 31 18:10 .
drwxr-xr-x  11 root sys      11 Dec 31 17:50 ..
-rw-------   1 tdh  nobody 5417 Dec 31 18:10 .tcshrc
[tdh@adept tdh]> grep 10 /etc/group
wheel:x:10:root

The nobody shows up for the group because there is no mapping between the string "staff" and "wheel". In NFSv3, the numeric 10 would have gone across the wire and the ls command would have spit out "wheel".

Okay, let's check to see what the Solaris client would have done:

[tdh@sandman ~]> ssh -fN -L "5049:wont:2049" wont
Password:
[tdh@sandman ~]> su -
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_54  October 2007
# mkdir -p /nfs4/wont
# mount -o port=5049 localhost:/ /nfs4/wont
# exit
[tdh@sandman ~]> cd /nfs4/wont
[tdh@sandman wont]> ls -la
total 134
drwxr-xr-x  38 root     root        1024 Dec 31 17:49 .
drwxr-xr-x   3 root     root         512 Dec 31 18:17 ..
...
drwxr-xr-x   2 root     root         512 Dec 31 14:51 Desktop
drwxr-xr-x   2 root     root         512 Dec 31 14:51 Documents
lrwxrwxrwx   1 root     root           9 Dec 31 13:17 bin -> ./usr/bin
drwxr-xr-x   5 root     sys          512 Dec 31 14:12 boot
drwxr-xr-x  24 root     sys         4096 Dec 31 14:42 dev
drwxr-xr-x  10 root     sys          512 Dec 31 14:42 devices
drwxr-xr-x  87 root     sys         4608 Dec 31 17:52 etc
drwxr-xr-x   4 root     sys          512 Dec 31 17:50 export
...
[tdh@sandman wont]> cd export
[tdh@sandman export]> ls -la
total 9
drwxr-xr-x   4 root     sys          512 Dec 31 17:50 .
drwxr-xr-x  38 root     root        1024 Dec 31 17:49 ..
drwxr-xr-x   2 root     root         512 Dec 31 13:17 home
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 zfs
[tdh@sandman export]> cd zfs
[tdh@sandman zfs]> ls -la
total 5
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 .
drwxr-xr-x   4 root     sys          512 Dec 31 17:50 ..

Okay, we have hit the crux of the problem for Mirror Mounts. We have a filesystem crossing on the server which needs to be mirrored on the client. We have to do this manually (or with an automounter if the ports are open):

[tdh@sandman zfs]> cd
[tdh@sandman ~]> su -
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_54  October 2007
# mount -o port=5049 localhost:/export/zfs /nfs4/wont/export/zfs
# ls -la /nfs4/wont/export/zfs
total 32
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 .
drwxr-xr-x   4 root     sys          512 Dec 31 17:50 ..
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 braves
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 kanigix
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 loghyr
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 mrx
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 nfsv2
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 nfsv3
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 nfsv4
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 spud
drwxr-xr-x   2 root     sys            3 Dec 31 18:10 tdh
# tcsh
# ls -la /nfs4/wont/export/zfs/tdh
total 6
drwxr-xr-x   2 root     sys            3 Dec 31 18:10 .
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 ..
# mount -o port=5049 localhost:/export/zfs/tdh /nfs4/wont/export/zfs/tdh
# ls -la  /nfs4/wont/export/zfs/tdh
total 18
drwxr-xr-x   2 root     sys            3 Dec 31 18:10 .
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 ..
-rw-------   1 tdh      staff       5417 Dec 31 18:10 .tcshrc

Notice how the '/export/zfs' gave information about the child filesystems whereas '/' did not. Also, note how we get the correct group name because the '/etc/group' is the same on the two Solaris hosts. Finally, even with zfs presenting up the child filesystems, we did have to manually mount the child in order to peer into it.

So the Mirror Mounts project in the NFSv4 development team is going to fix all of this. Under the hood, the client is going to understand it is about to traverse to a different filesystem and do the equivalent of a NFSv3 mount.


Technorati Tags:
Orginally posted on Kool Aid Served Daily
Copyright (C) 2006, Kool Aid Served Daily
About

tdh

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today