Friday - just a day late.

At Home

Friday is curry night - mmm! Dogs are tired, Finn seems to grow bigger everyday, he is only 5 months old but he is heavier and taller at the shoulder than Cleo who is a year older. If you want to see some Samoyeds have a peek at Samoyeds. To avoid causing a pet war here are the cats two-cats

At Work

So I got to the bottom of the busy cpu. That machine had a massive kernel device tree.  "prtconf -v | grep nstance | wc -l " was about 30,000.  Looking in the prtconf -v we see that 80% of the device nodes are "driver not attached" so they are  nodes without a driver instance bound to them. Surprisingly they are all for the same driver, time to investigate what is in that driver .conf file..

So what is the problem with having a big kernel device tree? Well  many utilities need to grab a snapshot of the tree in order to be able to match driver names/numbers/instances. On this machine iostat was one contributor and the other was the event gatherer for Sun's SRS product.  Grabbing the snapshot involves walking the in-kernel tree and copying it into a snapshot buffer, from there it can be copied out from the kernel to userland.
This is not a lightweight operation, the less often we make the kernel execute the more cpu cycles are available for the application to try and run its instructions.

To work out what process was issuing the ioctl, I could have trussed every application until I found the right one , but this machine has thousands of processes. So we used the tnf probes built into the kernel, luckily there are probes in the entry and return points for syscalls. So we turned those on and looked for the lines with system call number 54 (grep ioctl /etc/name_to_sysnum).

# prex -k
Type "help" for help ...
prex> buffer alloc 10m
Buffer of size 10485760 bytes allocated
prex> enable name=/syscall/
prex> trace name=/syscall/
prex> list probes name=/syscall/
name=syscall_end enable=on trace=on file=../../intel/ia32/os/syscall.c line=649 funcs=<no value>
name=syscall_start enable=on trace=on file=../../intel/ia32/os/syscall.c line=364 funcs=<no value>
prex> ktrace on
#wait a bit
prex> ktrace off
prex>quit
# tnfxtract /tmp/1.tnf
# tnfdump /tmp/1.tnf > /tmp/1.txt
# pg /tmp/1.txt  (and lets look for syscall 54)
---------------- ---------------- ----- ----- ---------- --- ------------------------- ------------------------
    Elapsed (ms)       Delta (ms)   PID LWPID    TID     CPU Probe Name                Data / Description . . .
---------------- ---------------- ----- ----- ---------- --- ------------------------- ------------------------
        0.113180         0.013207   719     1 0xffffffff839138a0   1 syscall_start             sysnum: 54
        0.113962         0.000782   719     1 0xffffffff839138a0   1 syscall_end               rval1: 0 rval2: 0 errno: 0
better dealloc that 10m buffer
# prex -k
prex> buffer dealloc
prex> quit
#

So it was pid 719, all we need now is ps -efl..

In Between

0 miles in the smart, but I had to fill it up with petrol - the massive amount of 16 litres. So that was 180 miles for 16 litres which is only 50 mpg. Letting the automatic mode change gear normally gets about 65 mpg but this tankful I have been using the manual mode as I thought I could do a better job, I will switch back as the point of getting a smart car was a cheap way to get to work.

15 miles on the bike.


tim.
Comments:

Post a Comment:
Comments are closed for this entry.
About

timatworkhomeandinbetween

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today
News

No bookmarks in folder

Blogroll

No bookmarks in folder