Wednesday Apr 08, 2009

Linux, and what we may think is RAW I/O

I had two instances this morning within one minute where Vdbench users were running into problems running high IOPS against what they thought were RAW Linux volumes.
1.7 million IOPS against a single disk. WOW!!!!!

Linux by default opens a /dev/sdx device as a cached device. When you run one TB of cached disk against 4GB of system memory, the OS (whether it is Linux or Solaris) falls apart. It runs out of memory, and every single CPU cycle available is used to find free memory. The end result is that your CPU runs at 100%, and almost everything hangs. Result: maybe you can not even log on, maybe you can not even cancel Vdbench, or maybe the Vdbench master JVM can not communicate with its slaves, with Vdbench therefore (if ever it gets some CPU cycles) running into heartbeat timeouts.

I am 100.000% sure that the objective of your tests is not to measure memory speeds, but instead to test the storage. To prevent Linux from opening a device as cached, you'll need to specify:
sd=sdx,lun=/dev/sdx,openflags=o_direct
or
sd=default,openflags=o_direct
sd=sd1,...
sd=sd2,....

Henk.

About

Blog for Henk Vandenbergh, author of Vdbench, and Sun StorageTek Workload Analysis Tool (Swat). This blog is used to keep you up to date about anything revolving around Swat and Vdbench.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today