Question: Why isn't a policy working?

One of the hardest things about developing software is testing it. As a developer, you get too close to the code and what you expect it to do. So I was very happy to get a question in email about why was a set of policies not working correctly:

I tried to reset the SPE policies to the following:

# cat /etc/policies.spe
101, 1, 8k, DS2, path == /diskpool/JUNK/TEST/P1
102, 2, 4k, DS1, path == /diskpool/JUNK/TEST/P2
103, 3, 2k, mix3, path == /diskpool/JUNK/TEST/P3
104, 4, 1k, mix4, path == /diskpool/JUNK/TEST/P4
#105, 5, 128k, default, path == /diskpool/JUNK/TEST/P5 || path == /export/JUNK
111, 5, 32k, default, path == /diskpool/JUNK/TEST/P5
112, 5, 64k, default, path == /export/JUNK

and have the client uses the path to verify the stripe.  For stripe 1,2,3
things seem working OK; however when I tried with strip 4 or 5 with
the policy above, I got "nfsstat -l" to list stripe 10 and count 32k:

pnfs-minipit1-3:~#
pnfs-minipit1-3:~# /mntpit1-3: mount -o vers=4 pnfs-minipit1-5:/diskpool/JUNK/TEST/P5 /mnt
pnfs-minipit1-3:~# nfsstat -m /mnt
/mnt from pnfs-minipit1-5:/diskpool/JUNK/TEST/P5
 Flags:         vers=4,proto=tcp,sec=sys,hard,intr,link,symlink,acl,rsize=1048576,wsize=1048576,retrans=5,timeo=600
 Attr cache:    acregmin=3,acregmax=60,acdirmin=30,acdirmax=60

pnfs-minipit1-3:~# mkfile -v 1m /mnt/kk
/mnt/kk 1048576 bytes
pnfs-minipit1-3:~# nfsstat -l /mnt/kk
Number of layouts: 1
Proxy I/O count: 0
DS I/O count: 33
Layout [0]:
        Layout obtained at: Sun Aug 30 00:23:47:199247 2009
        status: UNKNOWN, iomode: LAYOUTIOMODE_RW
        offset: 0, length: EOF
        num stripes: 10, stripe unit: 32768
        Stripe [0]:
                tcp:pnfs-minipit1-6.Central.Sun.COM:10.1.232.125:4920 OK
        Stripe [1]:
                tcp:pnfs-minipit1-6.Central.Sun.COM:10.1.232.125:4920 OK
        Stripe [2]:
                tcp:pnfs-minipit1-7.Central.Sun.COM:10.1.235.56:4920 OK
        Stripe [3]:
                tcp:pnfs-minipit1-7.Central.Sun.COM:10.1.235.56:4920 OK
        Stripe [4]:
                tcp:pnfs-minipit1-6.Central.Sun.COM:10.1.232.125:4920 OK
        Stripe [5]:
                tcp:pnfs-minipit1-6.Central.Sun.COM:10.1.232.125:4920 OK
        Stripe [6]:
                tcp:pnfs-minipit1-7.Central.Sun.COM:10.1.235.56:4920 OK
        Stripe [7]:
                tcp:pnfs-minipit1-7.Central.Sun.COM:10.1.235.56:4920 OK
        Stripe [8]:
                tcp:pnfs-minipit1-6.Central.Sun.COM:10.1.232.125:4920 OK
        Stripe [9]:
                tcp:pnfs-minipit1-6.Central.Sun.COM:10.1.232.125:4920 OK
pnfs-minipit1-3:~#

What did I do wrong?

Before we even look at the issue, what did the person asking for help do right

  1. Described the problem.
    1. What did they expect to see?
    2. What did they actually see?
  2. Showed me how to reproduce the problem.

In short, I'm confident that the person took all reasonable steps to solve the problem on their own.

So, now let's try to figure out what is going on. I have a DTrace script which will help out here. Ouch, I should have shared that earlier. Anyway, spe.d should help. I'll start it up and monitor the results as I do a file create:

[root@pnfs-minipit1-5 ~]> ./spe.d
dtrace: script './spe.d' matched 8 probes
CPU     ID                    FUNCTION:NAME
  1   3420 nfs41_spe_allocate:spe-i-check_open 200096 (10) from 10.1.233.191 is checking /diskpool/JUNK/TEST/P5/foollu2
  1   3419 nfs41_spe_allocate:spe-i-policy_eval Policy 101 did not match with error 0 from 10.1.233.191
  1   3419 nfs41_spe_allocate:spe-i-policy_eval Policy 102 did not match with error 0 from 10.1.233.191
  1   3419 nfs41_spe_allocate:spe-i-policy_eval Policy 103 did not match with error 0 from 10.1.233.191
  1   3419 nfs41_spe_allocate:spe-i-policy_eval Policy 104 did not match with error 0 from 10.1.233.191
  1   3419 nfs41_spe_allocate:spe-i-policy_eval Policy 111 matched with error 0 from 10.1.233.191
  1  63564    mds_ds_path_to_mds_sid:return mds_sid[0] = pnfs-minipit1-6:pNFSpool1/p1DS2
  1  63564    mds_ds_path_to_mds_sid:return mds_sid[1] = pnfs-minipit1-6:pNFSpool2/p2DS2
  1  63564    mds_ds_path_to_mds_sid:return mds_sid[2] = pnfs-minipit1-6:pNFSpool3/p3DS2
  1  63564    mds_ds_path_to_mds_sid:return mds_sid[3] = pnfs-minipit1-7:pNFSpool1/p1DS1
  1  63564    mds_ds_path_to_mds_sid:return ERROR - could not find a matching pgi for pnfs-minipit1-7:pNFSpool1/p2DS1
  1  56851        nfs41_spe_allocate:return No matching policy

So we did find a matching policy, but we only found 5 datasets and the policy requires 5. So, why isn't that dataset in memory?

[root@pnfs-minipit1-5 ~]> echo "::walk mds_DS_guid_entry_cache|::print struct rfs4_dbe dbe_data|::print ds_guid_info_t ds_dataset_name.utf8string_val|::eval /s" | mdb -k
0xffffff01e73e5fc0:             pnfs-minipit1-6:pNFSpool3/p3DS2�����f
0xffffff01e7227078:             pnfs-minipit1-6:pNFSpool3/p3DS1�����f
0xffffff01e7227120:             pnfs-minipit1-6:pNFSpool2/p2DS2�����f
0xffffff01e72271c8:             pnfs-minipit1-6:pNFSpool2/p2DS1�����f
0xffffff01e7227270:             pnfs-minipit1-6:pNFSpool1/p1DS2�����f
0xffffff01e7227318:             pnfs-minipit1-6:pNFSpool1/p1DS1�����f
0xffffff01e5d62e70:             pnfs-minipit1-7:pNFSpool2/p2DS2�����f
0xffffff01e6597628:             pnfs-minipit1-7:pNFSpool2/p2DS1�����f
0xffffff01e4e40468:             pnfs-minipit1-7:pNFSpool1/p1DS2�����f
0xffffff01e5f84cb8:             pnfs-minipit1-7:pNFSpool1/p1DS1�����f
[root@pnfs-minipit1-5 ~]> echo "::walk mds_DS_guid_entry_cache|::print struct rfs4_dbe dbe_invalid" | mdb -k
dbe_invalid = 0
dbe_invalid = 0
dbe_invalid = 0
dbe_invalid = 0
dbe_invalid = 0
dbe_invalid = 0
dbe_invalid = 0
dbe_invalid = 0
dbe_invalid = 0
dbe_invalid = 0 

Hmm, I need to figure out a better way to process UTF8 stings in mdb, but it appears to be there and in a valid entry. I couldn't figure this out until I lined up the dataset name output:

pnfs-minipit1-7:pNFSpool2/p2DS1
pnfs-minipit1-7:pNFSpool1/p2DS1

The naming convention shows the error, 'p2DS1' should be on pool #2, but it is on pool #1: pNFSpool1.

If this were implemented in SMF instead of a as a flat file, we would be able to do checking when we create the policy to catch this type of configuration error. But for now, DTrace and mdb can catch it.

For me, getting mdb to dump the dataset names was the hardest part. I had to experiment and search the web. I'm glad for all of the effort here, I'll use that trick again and again.


Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

tdh

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today