Find out process(es) with lwp(s)/thread(s) swapped out

test




I've been asked the question 'how do I find out what processes have been swapped out as I see non-zero number under kthr:w?' by our customers from time to time and I always answered him/her that it's not easy or just impossible.


I now can say that I'm wrong 'cause I've found the way to figure this out and in fact it's quite simple.


Here's my case.


I have a x4200 lab server which is runnning quite a few processes and jobs and I can see there're many(17) lwps being swapped-out


-----------
root@cdlabx4200-1[5.10]:/# vmstat 1 5
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr lf s0 s1 s2   in   sy   cs us sy id
 0 0 3 2330908 1113424 1  7  0  0  0  0 24  0 -0  2  2 3722 5646 3975  4  9 87
 1 0 17 1207200 162240 22 55 0  0  0  0  0  0  0  0  0 12939 23653 15759 19 40 41
 0 0 17 1207112 162208 2  4  0  0  0  0  0  0  0  0  0 12710 23441 15445 19 40 41
 0 0 17 1207112 162208 2  2  0  0  0  0  0  0  0  0  0 13589 25916 16710 20 41 39
 1 0 17 1207112 162208 2  2  0  0  0  0  0  0  0  0  0 13738 25857 16785 20 40 40
root@cdlabx4200-1[5.10]:/# 
------------------
manpage of vmstat
-----------------
     kthr            Report the number of kernel threads in  each
                     of the three following states:
                     w        the number of  swapped  out  light-
                              weight  processes  (LWPs)  that are
                              waiting for processing resources to
                              finish.
-----------------
So my question is what are these 17 lwps or in another way what processes contain these 17 lwps.
Unfortunately neither ps nor prstat provides simple method/option to get this info.
'-l' option in ps shows us detailed information of a process, including below state:
---------------------
     S (l)               The state of the process:
                         O            Process  is  running  on  a
                                      processor.
                         S            Sleeping: process is  wait-
                                      ing  for  an  event to com-
                                      plete.
                         R            Runnable: process is on run
                                      queue.
                         T            Process is stopped,  either
                                      by  a job control signal or
                                      because it is being traced.
                         W            Waiting: process is waiting
                                      for  CPU  usage  to drop to
                                      the CPU-caps enforced  lim-
                                      its.
                         Z            Zombie state: process  ter-
                                      minated   and   parent  not
                                      waiting.
----------------------------
which one corresponds to 'kthr:w'? None. As I aggregated all S column in 'ps -eflL' output
---------------
root@cdlabx4200-1[5.10]:/# ps -eflL|awk '{print $2}'|sort|uniq -c
   5 O
1974 S
   1 T
  14 Z
---------------
 no '17'.
 how about 'STATE' column in prstat -L?
----------------
root@cdlabx4200-1[5.10]:/# prstat -cLn3000 1 1 |awk '{print $5}'|sort|uniq -c
   1 STATE
   1 cpu0
   1 cpu1
   1 cpu2
   1 lwps,
1971 sleep
  14 zombie
----------------
no clue either.
Well, ps and prstat report only active processes while swapped lwps are not active at all.
What else? Does mdb help?
Every process has in-kernel a 'proc_t' structure associated with it and check if there's any clue
-----------------------
root@cdlabx4200-1[5.10]:/# mdb -k
Loading modules: [ unix krtld genunix dtrace specfs uppc pcplusmp cpu.generic cpu_ms.AuthenticAMD.15 zfs mpt sockfs ip hook net
i sctp arp usba fcp fctl qlc nca lofs md cpc fcip random crypto logindmux ptm ufs sppp nfs sata ipc ]
::print -t proc_t!grep swap


int p_swapcnt
u_longlong_t nswap 
u_longlong_t nswap 


------------------------------
looks i'm lucky, there's member 'p_swapcnt' which should have something to do with swapped lwps.


check if there's any comment in source code (RTFS)
---------------------
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/sys/proc.h#149
149 intp_swapcnt;/* number of swapped out lwps */
----------------------
This should be what I'm looking for.
Next step is simple, just to walk through the process list and print out p_swapcnt in mdb
-------------------
::walk proc|::print -t proc_t p_swapcnt
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0
int p_swapcnt = 0x1  <--- at least one
int p_swapcnt = 0
...
------------------
I trimmed the output for simplicity. At least i got one.(Note that p_swapcnt is in base 16)
Well i want to print out pids with their p_swapcnt. It's a simple task in awk.
------------------------
root@cdlabx4200-1[5.10]:/# echo '::walk proc|::print -t proc_t p_pidp->pid_id p_swapcnt'|\
mdb -k|awk '{if(NR%2){printf("%s\t",$0);}else{printf("%s\n",$0);}}'|\
awk '{if($NF!=0){printf("pid: %s\tp_swapcnt: %s\n",$4,$NF);}}'

pid: 0x102      p_swapcnt: 0x1
pid: 0x601d     p_swapcnt: 0x1
pid: 0x7d1      p_swapcnt: 0xf
root@cdlabx4200-1[5.10]:/# 
---------------------------
The sum of all p_swapcnt is just 17
--------------------------
root@cdlabx4200-1[5.10]:/# mdb -k
Loading modules: [ unix krtld genunix dtrace specfs uppc pcplusmp cpu.generic cpu_ms.AuthenticAMD.15 zfs mpt sockfs ip hook net
i sctp arp usba fcp fctl qlc nca lofs md cpc fcip random crypto logindmux ptm ufs sppp nfs sata ipc ]
> 1+1+f=d
                17      

----------------------
Bingo, I got the culprits.
It's easy to convert pid to decimal.
-----------------
> 0x102=d
                258     
> 0x601d=d
                24605   
> 0x7d1=d
                2001    

------------------
-EOF- 





评论:

I have begun using (based in part on How to use DTrace and mdb to Interpret vmstat Statistics [ID 1009494.1]):

echo "::walk thread thr |::print kthread_t t_schedflag|::grep .==0 |::eval <thr=K|::print -d kthread_t t_procp->p_pidp->pid_id t_procp->p_user.u_comm" | mdb -k

It has the advantage of showing the pid in base 10 as well as the command that started it.

thanks for the info,
GlenG

发表于 guest 在 2012年02月14日, 02:55 上午 CST #

Hi GlenG,

That's great.

thanks

Sam

发表于 Sam Wan 在 2012年03月06日, 11:33 上午 CST #

发表一条评论:
  • HTML语法: 禁用
About

samwan

Search

Categories
Archives
« 四月 2014
星期日星期一星期二星期三星期四星期五星期六
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
今天