星期三 二月 08, 2012

Does swapped-out LWP have 't_schedflag==0'?

In last post, I use p_swapcnt info in proc_t to find out process(es) with lwp(s)/thread(s) swapped-out.

I also tried to walk through threads list and check their flags/states, however I'm not quite sure which one corresponds to the swapped-out state.

I do find that if I check on t_schedflag, when it's '0', the pids are the same as what I get from last post

-----------------------

echo "::walk thread|::print -t kthread_t t_tid t_schedflag t_procp->p_pidp->pid_id"|mdb -k > /tmp/walk_thread.out

-----------------------

I then use a awk script to print out those threads whose t_schedflag is 0

--------------------------


root@cdlabx4200-1[5.10]:/# awk '{if(NR%3){printf("%s\t",$0);}else{printf("%s\n",$0);}}' /tmp/walk_thread.out |awk '{if($8==0){printf("pid=%s\ttid=%4s\tt_schedflag=%s\n",$NF,$4,$8);}}'
pid=0x102 tid= 0x2 t_schedflag=0
pid=0x7d1 tid= 0x2 t_schedflag=0
pid=0x7d1 tid= 0x3 t_schedflag=0
pid=0x7d1 tid= 0x4 t_schedflag=0
pid=0x7d1 tid= 0x5 t_schedflag=0
pid=0x7d1 tid= 0x6 t_schedflag=0
pid=0x7d1 tid= 0x7 t_schedflag=0
pid=0x7d1 tid= 0xa t_schedflag=0
pid=0x7d1 tid= 0xc t_schedflag=0
pid=0x7d1 tid= 0xe t_schedflag=0
pid=0x7d1 tid= 0xf t_schedflag=0
pid=0x7d1 tid=0x10 t_schedflag=0
pid=0x7d1 tid=0x11 t_schedflag=0
pid=0x7d1 tid=0x12 t_schedflag=0
pid=0x7d1 tid=0x13 t_schedflag=0
pid=0x7d1 tid=0x14 t_schedflag=0
pid=0x601d tid= 0x1 t_schedflag=0
root@cdlabx4200-1[5.10]:/#


--------------------------------


See pids are the same as in last post.


I guess when a thread is swapped out, its t_schedflag is cleared because it makes no sense any more.


-EOF-



星期二 二月 07, 2012

Find out process(es) with lwp(s)/thread(s) swapped out

test




I've been asked the question 'how do I find out what processes have been swapped out as I see non-zero number under kthr:w?' by our customers from time to time and I always answered him/her that it's not easy or just impossible.


I now can say that I'm wrong 'cause I've found the way to figure this out and in fact it's quite simple.


Here's my case.


I have a x4200 lab server which is runnning quite a few processes and jobs and I can see there're many(17) lwps being swapped-out


-----------
root@cdlabx4200-1[5.10]:/# vmstat 1 5
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr lf s0 s1 s2   in   sy   cs us sy id
 0 0 3 2330908 1113424 1  7  0  0  0  0 24  0 -0  2  2 3722 5646 3975  4  9 87
 1 0 17 1207200 162240 22 55 0  0  0  0  0  0  0  0  0 12939 23653 15759 19 40 41
 0 0 17 1207112 162208 2  4  0  0  0  0  0  0  0  0  0 12710 23441 15445 19 40 41
 0 0 17 1207112 162208 2  2  0  0  0  0  0  0  0  0  0 13589 25916 16710 20 41 39
 1 0 17 1207112 162208 2  2  0  0  0  0  0  0  0  0  0 13738 25857 16785 20 40 40
root@cdlabx4200-1[5.10]:/# 
------------------
manpage of vmstat
-----------------
     kthr            Report the number of kernel threads in  each
                     of the three following states:
                     w        the number of  swapped  out  light-
                              weight  processes  (LWPs)  that are
                              waiting for processing resources to
                              finish.
-----------------
So my question is what are these 17 lwps or in another way what processes contain these 17 lwps.
Unfortunately neither ps nor prstat provides simple method/option to get this info.
'-l' option in ps shows us detailed information of a process, including below state:
---------------------
     S (l)               The state of the process:
                         O            Process  is  running  on  a
                                      processor.
                         S            Sleeping: process is  wait-
                                      ing  for  an  event to com-
                                      plete.
                         R            Runnable: process is on run
                                      queue.
                         T            Process is stopped,  either
                                      by  a job control signal or
                                      because it is being traced.
                         W            Waiting: process is waiting
                                      for  CPU  usage  to drop to
                                      the CPU-caps enforced  lim-
                                      its.
                         Z            Zombie state: process  ter-
                                      minated   and   parent  not
                                      waiting.
----------------------------
which one corresponds to 'kthr:w'? None. As I aggregated all S column in 'ps -eflL' output
---------------
root@cdlabx4200-1[5.10]:/# ps -eflL|awk '{print $2}'|sort|uniq -c
   5 O
1974 S
   1 T
  14 Z
---------------
 no '17'.
 how about 'STATE' column in prstat -L?
----------------
root@cdlabx4200-1[5.10]:/# prstat -cLn3000 1 1 |awk '{print $5}'|sort|uniq -c
   1 STATE
   1 cpu0
   1 cpu1
   1 cpu2
   1 lwps,
1971 sleep
  14 zombie
----------------
no clue either.
Well, ps and prstat report only active processes while swapped lwps are not active at all.
What else? Does mdb help?
Every process has in-kernel a 'proc_t' structure associated with it and check if there's any clue
-----------------------
root@cdlabx4200-1[5.10]:/# mdb -k
Loading modules: [ unix krtld genunix dtrace specfs uppc pcplusmp cpu.generic cpu_ms.AuthenticAMD.15 zfs mpt sockfs ip hook net
i sctp arp usba fcp fctl qlc nca lofs md cpc fcip random crypto logindmux ptm ufs sppp nfs sata ipc ]
::print -t proc_t!grep swap


int p_swapcnt
u_longlong_t nswap 
u_longlong_t nswap 


------------------------------
looks i'm lucky, there's member 'p_swapcnt' which should have something to do with swapped lwps.


check if there's any comment in source code (RTFS)
---------------------
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/sys/proc.h#149
149 intp_swapcnt;/* number of swapped out lwps */
----------------------
This should be what I'm looking for.
Next step is simple, just to walk through the process list and print out p_swapcnt in mdb
-------------------
::walk proc|::print -t proc_t p_swapcnt
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0x1  <--- at least one
int p_swapcnt = 0
...
------------------
I trimmed the output for simplicity. At least i got one.(Note that p_swapcnt is in base 16)
Well i want to print out pids with their p_swapcnt. It's a simple task in awk.
------------------------
root@cdlabx4200-1[5.10]:/# echo '::walk proc|::print -t proc_t p_pidp->pid_id p_swapcnt'|\
mdb -k|awk '{if(NR%2){printf("%s\t",$0);}else{printf("%s\n",$0);}}'|\
awk '{if($NF!=0){printf("pid: %s\tp_swapcnt: %s\n",$4,$NF);}}'

pid: 0x102      p_swapcnt: 0x1
pid: 0x601d     p_swapcnt: 0x1
pid: 0x7d1      p_swapcnt: 0xf
root@cdlabx4200-1[5.10]:/# 
---------------------------
The sum of all p_swapcnt is just 17
--------------------------
root@cdlabx4200-1[5.10]:/# mdb -k
Loading modules: [ unix krtld genunix dtrace specfs uppc pcplusmp cpu.generic cpu_ms.AuthenticAMD.15 zfs mpt sockfs ip hook net
i sctp arp usba fcp fctl qlc nca lofs md cpc fcip random crypto logindmux ptm ufs sppp nfs sata ipc ]
> 1+1+f=d
                17      

----------------------
Bingo, I got the culprits.
It's easy to convert pid to decimal.
-----------------
> 0x102=d
                258     
> 0x601d=d
                24605   
> 0x7d1=d
                2001    

------------------
-EOF- 





About

samwan

Search

Categories
Archives
« 二月 2012 »
星期日星期一星期二星期三星期四星期五星期六
   
1
2
3
4
5
6
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
   
       
今天