Monday Sep 23, 2013

Using Analytics (Dtrace) to Troubleshoot a Performance Problem, Realtime

I was recently on a call with one of the largest financial retirement investment firms in the USA.  They were using a very small 7320 ZFS Storage appliance (ZFSSA) with 24GB of dram and 18 x 3TB HDDs.  This system was nothing in terms of specifications compared to the latest shipping systems, the  ZS3-2 and ZS3-4 storage systems with 512GB and 2TB of DRAM respectively.  Nevertheless I was more then happy to see what help myself and mostly the ZFSSA Analytics (Dtrace) could offer.  

The Problem: Umm, Performance aka Storage IO Latency is less then acceptable...

This customer was/is using this very small ZFSSA as an additional Data Guard target from their production Exadata.  In this case it was causing issues and pain.  Below you can see a small sample screen shot we got from their Enterprise Manager Console.

Discovery: Lets see what is going on right now! Its time to take a little Dtrace tour..

The next step was to fire up the browser and take a look at the real-time analytic's.    Our first stop on the dtrace train is to look at IOPS by protocol.  In this case the protocol is NFSv3 but we could easily see the same ting from nfsv4, smb, http, ftp, fc, iscsi etc... Quickly we see that this little box with 24GB of DRAM and 18 x 7200RPM HDD's was being pounded! Averaging 13,000 IOPS with 18 drives isn't bad.  I should note that this box had zero Read SSD drives and 2 x Write SSD drives.  

Doing a quick little calculation we see that this little system is doing about 650 IOPS per drive per second. Holy Cow SON! Wait, is that even possible? There is something else at play here, could it be the large ZFSSA cache (a measly 24GB in our case) at work?  Hold your horses, IOPS are not everything in fact you could argue that they really don't matter at all, what really matters is latency.  This is how we truly measure performance, how fast does it take to do one thing versus how many things can I do at the same time regardless of how fast they each get done.  To best understand how much effect DRAM has on latency see our World Record SPECSFS Benchmark information here.  Here you see that sure enough that some of the read latency is pathetic, for just about any application in the world.  

There are many ways to solve this problem, the average SAN/NAS vendor would tell you to simply add more disk, With dtrace we can get more granular and ask many other questions and see if there is perhaps a better or more efficient way(s) to solve this problem.

This leads us to our next stop on the dtrace discovery train, ARC accesses (Adaptive Replacement Cache).  Here we quickly find that even with our lackluster performance in terms of read latency.  We have an amazing read cache hit ratio.  Roughly about 60% of our IO is coming from our 24GB of DRAM. On this particular system the DRAM is upgradable to 144GB of DRAM.  Do ya think that would make a small dent in those 6033 data misses below?

This nicely leads into the next stop on the dtrace train which is to ask dtrace for all those 6033 data misses how many of them would be eligible to be read from L2ARC (READ SSD Drives in the Hybrid Storage Pool).  We quickly noticed that indeed they would have made a huge difference.  Sometimes 100% of the misses were eligible.  This means that after missing the soon to be upgraded 6x dram based cache the rest of the read IO's of this workload would likely be served from high performance MLC Flash SSD right in the controller itself.

Conclusion: Analytics on the ZFSSA are amazing, The Hybrid Storage Pool of the ZFSSA is amazing, the large DRAM based cache on the ZFSSA is very amazing...

At this point I recommend that they take a two phase approach to the workload.  First they upgrade the DRAM Cache 6x and add 2 x L2ARC Read SSD drives.  After that they could evaluate if they still needed to add more disk or not.

Extra Credit Stop:  One last stop I made  was to look at their NFS share settings and see if they had compression turned on like I have recommended in a previous post.  I noticed that they did not have it enabled and that CPU was very low at less then 10% AVG utilization.  I then explained to the customer how they would benefit even now without any upgrades by enabling compression and I asked if I could enable it that second at 12pm in the afternoon during a busy work day.  They trusted me and we enabled it on the fly and then for giggles I decided to see if it made a difference on disk IO and sure enough it made an immediate impact and disk IO lowered because now every new write they had was taking less disk space and less IO to move through the subsystem since we compress real-time in memory.  Remember that this system was a Data guard target so it was constantly being written too. You can clearly see in the image below how compression lowered the amount of  IO on the actual drives.  Trying doing that with your average storage array compression.

Special Thanks goes out to my teammates Michael Dautle and Hugh Shannon who helped with this adventure and capturing of the screenshots.

Friday Jul 13, 2012

Oracle ZFSSA Smashes IBM XIV While Running Oracle ERP On Oracle RAC DB

I recently was part of a customer proof of concept where we tested running their Oracle ERP on Oracle T-4 servers along with a Oracle RAC on T-4's and storage was an Oracle ZFS Storage Appliance 7420.   The performance we achieved was awesome!  Without any optimization we were 2.5x faster then their existing production system on running their month end close.

Big Ugly SQL Query Problems:

The DBA also ran what he called some ugly queries they run and compared the results.  Again we smoked the XIV.

How Did You Do This?

The XIV had 144 7200 RPM drives we had 60 15k Drives.  The answer is CACHE, and lots of it...  The system we used for the test had 1TB of cache split on 2 controllers, 2 TB of L2arc Read SSDs and a few write SSDs to boot! Our hybrid storage pool design accounts for a lot of the high performance.  Keeping the active IO in a higher tier significantly lowered disk IO and in return brought down the latencies....  Notice the incredible low latencies, out of about 11000 IOPS over 10000 of them are less then 1.02ms.  When we looked at the AWR report from the previous month end close the number one wait event was "db file sequential read" with an average wait of 14ms and attributed to 50% of the top 5 wait events.  I would say we made a huge dent in that analytic.

What was interesting was that we really only used about 25% of what the total hardware setup could do.  We realized application modification and parallelization could produce a much larger performance improvement.  Unfortunately we were out of time on this POC but the customer is expecting to see even larger gains in performance in the future.  

What Else Did You See?

Interestingly we turned on lzjb compression which uses almost no CPU on our controllers to see what type of compression we could get on their database.  We saw 3.28x compression on their database data which was awesome!  This is more goodness for IO.  This means we transfer 3.28x less data back and forth to the slowest component in the whole system, spinning disk.  The customer also could expect to buy much less disk.  In a test/dev environment they could use our integrated snapshot/cloning feature to quickly make many test/dev copies of the database without worry of running out of disk space.  It certainly was a win-win feature.

What Does It All Mean? 

  1. The Oracle ZFSSA is a very fast platform to run Oracle Applications and Databases on.
  2. The Oracle ZFSSA is a solid platform and well tested within Oracle and outside of Oracle for Oracle Applications and Databases. (We had zero problems or errors in our POC)
  3. The Oracle ZFSSA Dtrace Analytics bring rich performance reporting to the storage device.
  4. The Oracle ZFSSA Compression features both improve performance and save customers money.
  5. When sizing a NAS/SAN device to run Oracle, CACHE matters.... The ZFSSA Hybrid Storage Pool has one of the most advanced Cache systems around.


Various information about Oracle Storage.


« July 2016