Capacity versus Performance

When I started this blog I said that I did performance and capacity planning for Sun's customers. I want to offer up a technical study or two to help others with performance issues. I entitled this Capacity vs Performance in order to highlight the difference. Often a capacity limitation manifests itself as a performance issue. In order to differentiate between performance and capacity, performance might be defined as 'How fast it is going' while capacity is 'the maximum performance of the system or an individual component.' Imagine capacity as the dump truck carrying a load and performance as a sports car racing. Even a sports car has to slow down for corners. Not to be too simple but we need to look at each component of the system's performance, CPU, memory, network, disk and tape.

One specific example was a customer who has a directory on the internet. Their customers submit searches from multiple sites and the Service Level Agreement (SLA) was no more than 5% of requests with response times of over 3 seconds. Currently 15% of request take more than 3 seconds which puts our customer in a penalty situation. The system is a 6800 with 12x900MHz CPUs. Unfortunately someone attempted to fix the problem by 'throwing more iron' at it and adding CPUs and memory without knowing why there was a problem. Lets look at a few numbers. From vmstat:

 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m1 m1   in   sy   cs us sy id
 0 2 0 8948920 5015176 374 642 10 12 13 0 2 1  2  1  2  132 2694 1315 14  3 83
 0 19 0 4089432 188224 466 474 50 276 278 0 55 5 5 4 3 7033 6191 2198 19  4 77
 0 19 0 4089232 188304 430 529 91 211 211 0 34 8 6 5 4 6956 9611 2377 16  5 79
 0 18 0 4085680 188168 556 758 96 218 217 0 40 12 4 6 4 6979 7659 2354 18 6 77
 0 18 0 4077656 188128 520 501 75 217 216 0 46 9 3 5 2 7044 8044 2188 17  5 78

There is something odd about these numbers. On vmstat, we look at the right 3 columns, us=user, sy=system and id=idle, so there is over 50% idle CPU available to throw at the problem. One way to detect a memory problem is to look at the sr, Scan Rate, column of vmstat (near the middle of the display.) If the page scanner ever starts running, or sr gets over 0, then we need to dig deeper into the memory system. The very odd part of this display is that the blocked queue on the left of the display has 18 or 19 processes in it but there are no processes in the run queue. That means we are blocking somewhere in Solaris without using all the CPUs available to us.

So now, we need to turn to the I/O subsystem. With Solaris 8, the iostat command has a new switch, -C which will aggregate I/Os at the controller level. My favorite iostat command is iostat -xnMCz -T d (interval in seconds) (count of iostat outputs):

                    extended device statistics              
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
  396.4   10.7    6.6    0.1  0.0 20.3    0.0   49.9   0 199 c1
  400.2    8.8    6.7    0.0  0.0 20.2    0.0   49.4   0 199 c3
  199.3    6.0    3.3    0.0  0.0 10.1    0.0   49.4   0  99 c1t0d0
  197.1    4.7    3.3    0.0  0.0 10.2    0.0   50.4   0 100 c1t1d0
  198.2    3.7    3.4    0.0  0.0  9.4    0.0   46.3   0  99 c3t0d0
  202.0    5.1    3.3    0.0  0.0 10.8    0.0   52.4   0 100 c3t1d0

Whoa! On controller 1 we are doing 396 reads per second and on controller 3 we are doing 400 reads per second. On the right side of the data we see that iostat thinks the controller is almost 200% busy (iostat error...never checked to see if there has been a bug filed.) So then the individual disks are doing almost 200 reads per second and iostat figures thats 100% busy on the disks. That leads us to a rule of thumb or hueristic, that individual disks perform at approximately 150 I/Os per second. This does not apply to LUNs or LDEVs from the big disk arrays. So our examination of the numbers lets us suggest adding 2 disks to each controller and relaying out the data. Unfortunately, due to the disk array configurations, we could only add 1 disk to each controller. That did improve the situation as seen by the next iostat:

                    extended device statistics              
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
  410.6    5.4    4.8    0.0  0.0  5.7    0.0   13.7   0 218 c1
  386.0    9.0    4.6    0.0  0.0  5.3    0.0   13.4   0 211 c3
  129.4    2.2    1.5    0.0  0.0  1.9    0.0   14.7   0  73 c1t0d0
  139.4    1.8    1.6    0.0  0.0  2.3    0.0   16.0   0  79 c1t1d0
  141.8    1.4    1.7    0.0  0.0  1.5    0.0   10.4   0  66 c1t2d0
  133.0    1.0    1.6    0.0  0.0  2.1    0.0   15.6   0  76 c3t0d0
  125.4    2.2    1.5    0.0  0.0  1.9    0.0   14.6   0  72 c3t1d0
  127.6    5.8    1.5    0.0  0.0  1.4    0.0   10.2   0  63 c3t2d0

We are still close to the top end of the performance of an individual disk but we dropped from 15% of transactions out of the SLA down to 6 or 7% of transactions out of the SLA. And the CPUs look good:

 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m1 m1   in   sy   cs us sy id
 0 2 0 9283064 5482928 787 1293 36 0 0 0 0  0 23  0 13 5145 14763 1394 27 6 67
 0 1 0 6547512 2483056 869 984 110 0 0 0 0  0 14  0  8 5377 8114 1372 23  6 71
 0 1 0 6525816 2461496 1190 1230 0 0 0 0 0  0  0  0  0 6414 17808 1402 33 9 58
 0 1 0 6516240 2451976 1316 481 0 0 0 0  0  0  0  0  0 5432 8226 1509 30  7 63
 0 1 0 6506616 2442768 684 660 0 0 0  0  0  0  0  0  0 5188 16922 1259 26 7 67

Now we still have plenty of idle CPU time and only 1 or 2 in the blocked queue. It would have been nice to be able to add 2 disks to each controller but even 1 disk on each alleviated this problem. After this, the customer studied some of the internal design of their directory search algorithms. As the proverb says, Fixing one performance or capacity limitation only exposes the next issue.

The point of this exercise is looking at all the numbers and attempting to locate the precise nature of the problem. Do not assume adding CPUs and memory will fix all peformance problems. In this case the search programs were exceeding the capacity of the disk drives which manifested itself as a performance problem of transactions with extreme response times. All those CPUs were waiting on the disk drives. One other thing to note in this example is that there were no 'magic' /etc/system parameters to tweak. There are fewer and fewer knobs (or parameters) in Solaris to adjust to improve performance.

Comments:

Great little post. More stories like these with different scenarios!

Posted by Kenneth on tetor 19, 2004 at 12:11 PD PDT #

Nice post! This kind of information through real world examples is very helpful to a budding capacity/performance guy like myself. Any other "rule of thumb" metrics like the one mentioned about "150 I/O's per disk"? Thanks again!

Posted by Jason on dhjetor 06, 2004 at 10:08 MD PST #

Post a Comment:
Comments are closed for this entry.
About

pcr

Search

Archives
« maj 2015
DieHënMarMërEnjPreSht
     
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
      
Today