Linux, and what we may think is RAW I/O
By Henk Vandenbergh-Oracle on Apr 08, 2009
I had two instances this morning within one minute where Vdbench users were running into problems running high IOPS against what they thought were RAW Linux volumes.
1.7 million IOPS against a single disk. WOW!!!!!
Linux by default opens a /dev/sdx device as a cached device. When you run one TB of cached disk against 4GB of system memory, the OS (whether it is Linux or Solaris) falls apart. It runs out of memory, and every single CPU cycle available is used to find free memory. The end result is that your CPU runs at 100%, and almost everything hangs. Result: maybe you can not even log on, maybe you can not even cancel Vdbench, or maybe the Vdbench master JVM can not communicate with its slaves, with Vdbench therefore (if ever it gets some CPU cycles) running into heartbeat timeouts.
I am 100.000% sure that the objective of your tests is not to measure memory speeds, but instead to test the storage. To prevent Linux from opening a device as cached, you'll need to specify: