About a week ago a developer brought to our attention a dilemma, a small 1.5 GB table was executing FTS much faster on his test instance than on the development instance.
Test is a 10g single instance on vxfs, on a 4 cpu v40 with 16GB of memory, running RH4 32 bits.
Dev is a 10g RAC with ASM and each instance is also on a 4 cpu v40 with 16GB of memory, running RH4, in this case 64 bits.
We devoted a lot of time to benchmark performance of FTS on both systems, and also comparing performance of ASM vs FS on the same RAC.
Small to medium sized tables were finishing FTS faster on FS than on ASM. Large to very large tables finished faster on ASM.
Insert was faster on ASM.
But we expected to have faster FTS on ASM in all tests. The investigation led us to a report that stated 'FTS on FS faster than ASM'. The main issue described on the report was a problem of configuration, ASM was configured with only one fiber card and with block devices instead of raw devices.
Once this was fixed FTS on ASM performed as well or faster than on FS.
Based on this information we started an end to end review of our configuration, we also had a single fiber card on the nodes of this RAC.
Take a look at the RAC Detail Design and Best Practices
Production and Development systems should not compromise on best practices, they can be, as in this case, critical to performance and stability.