Wednesday Mar 21, 2012

Using all Ten IO slots on a 7420

So I had the opportunity recently to actually use up all ten slots in a clustered 7420 system. This actually uses 20 slots, or 22 if you count the clusteron card. I thought it was interesting enough to share here. This is at one of my clients here in southern California.

You can see the picture below. We have four SAS HBAs instead of the usual two. This is becuase we wanted to split up the back-end taffic for different workloads. We have a set of disk trays coming from two SAS cards for nothing but Exadata backups. Then, we have a different set of disk trays coming off of the other two SAS cards for non-Exadata workloads, such as regular user file storage. 
We have 2 Infiniband cards which allow us to do a full mesh directly into the back of the nearby, production Exadata, specifically for fast backups and restores over IB. You can see a 3rd IB card here, which is going to be connected to a non-production Exadata for slower backups and restores from it.
The 10Gig card is for client connectivity, allowing other, non-Exadata Oracle databases to make use of the many snapshots and clones that can now be created using the RMAN copies from the original production database coming off the Exadata. This allows for a good number of test and development Oracle databases to use these clones without effecting performance of the Exadata at all.
We also have a couple FC HBAs, both for NDMP backups to an Oracle/StorageTek tape library and also for FC clients to come in and use some storage on the 7420.

 Now, if you are adding more cards to your 7420, be aware of which cards you can place in which slots. See the bottom graphic just below the photo. 
Note that the slots are numbered 0-4 for the first 5 cards, then the "C" slots which is the dedicated Cluster card (called the Clustron), and then another 5 slots numbered 5-9.

Some rules for the slots:

  • Slots 1 & 8 are automatically populated with the two default SAS cards. The only other slots you can add SAS cards to are 2 & 7.
  • Slots 0 and 9 can only hold FC cards. Nothing else. So if you have four SAS cards, you are now down to only four more slots for your 10Gig and IB cards. Be sure not to waste one of these slots on a FC card, which can go into 0 or 9, instead. 
  • If at all possible, slots should be populated in this order: 9, 0, 7, 2, 6, 3, 5, 4

Monday Mar 12, 2012

Good papers and links for the ZFSSA

So I have a pretty good collection of links and papers for the ZFSSA, and instead of giving them out one-at-a-time when asked, I thought it may be easier to do it this way. Many of the links from my old blog last May no longer work, so here is an updated list of some good spots to check out.

These are for ZFS, in general, not the ZFSSA, but it gives one good insight to how ZFS functions:

Tuesday Mar 06, 2012

New 7420 hardware released today

Some great new upgrades to the 7420 were announced and released today. You can now get 10-core CPUs in your 7420, allowing you to have 40 cores in each controller. Even better, you can now also go to a huge 1TB of DRAM for your L1ARC in each controller, using the new 16GB DRAM modules.

So your new choices for the new 7420 hardware are 4 x 8-core or 4 x 10-core models. Oracle is no longer going to sell the 2 x CPU models, and they are also going to stop selling the 6-core CPUs, both as of May 31st. Also, you can now order 8GB or 16GB modules, meaning that the minimum amount of memory is now 128GB, and can go to 1TB in each controller. No more 64GB, as the 4GB module has also been phased out (starting today, actually).

Now before you get upset that you can no longer get the 2-CPU model, be aware that there was also a price drop, so that the 4 x 8-core CPU model is a tad LESS then the old 2 x 8-core CPU model. So stop complaining.

It's the DRAM that I'm most excited about. I don't have a single ZFSSA client that I know of that has a CPU bottleneck. So the extra cores are great, but not amazing. What I really like is that my L1ARC can now be a whole 1TB. That's crazy, and will be able to drive some fantastic workloads. I can now place your whole, say 800GB, database entirely in DRAM cache, and not even have to go to the L2ARC on SSDs in order to hit 99% of your reads. That's sweet. 

Friday Jan 20, 2012

New Storage Magazine awards for NAS... Check this out...

Well, it's hard to be quiet about this. Storage Magazine just came out with the January 2012 issue, showing Oracle Storage doing quite well (#1) with the Oracle ZFSSA 7420 and 7320 family. Check out pages 37-43 of this month's Storage Magazine.

Storage Magazine: (pages 37-43)


Friday Sep 16, 2011

Resetting your 7420 password

So, you have a 7420 demo or test system, and either forgot which password you used or it came from another department or company and it still has a password.

You're up a creek, right?

No, there is a way to fix it. We had to do this with a demo box that was wiped clean of all data, but still had a non-standard password on it. We could have sent it back for the demo pool engineers to fix, but then we would have had to wait a week or more to start testing. So here is a document I made with the steps I took to fix both the ILOM password and the Fishwork Appliance Kit password on this 7420.

Disclaimer--- Yes, physical security is VERY IMPORTANT. If someone can touch your system, they can do this. Of course, if someone can touch your system, they can also unplug it, remove the drives, or un-rackit and take it, couldn't they?

Tuesday Aug 02, 2011

Important - Clean your OS drives. Before you update your systems.

Some folks out there are not reading the readme file before they update, and are running into trouble. Remember the "Important Safety Tip" line in Ghostbusters about not crossing the streams? This is kind of like that.

Have you cleaned up your OS drives lately?

It's important. You should do this every now and then, but especially before you do a big update, such as the update to Q3.4 (which may have you running multiple updates, as explained in my previous post)

You want to remove system updates that are getting old. You may want to keep the previous system software in order to roll back, but do you really need the one before that and the one before that? Are you really going to roll back your 7000 to the system code you used in September 2010? I doubt it. Don't be a hoarder. Delete it. Call Dr. Drew if you need to.
Then, let's check your analytic datasets. These can get big before you know it. If you leave them running all the time, even when you don't need to collect data, you're asking for trouble. Either only run them when you need to collect data (you can do this manually or via a script or alert), or for goodness sake export them as a .CSV file at the end of each month and delete them off the system so they don't get too large. There have been issues reported of problems with the systems once these files got much too large. Part of those issues have been addressed in Q3.4, but you still need to keep it clean.

Very bad things will happen if you let your OS drives fill up. Think of all the atoms of your body moving away from each other at the speed of light. Furthermore, when you do an update, the update will go much more smoothly if you clean up the OS drives, first.
See the readme section below, which I'm sure you forgot to read...

The following section comes from the Q3.4 readme file:

1.4 System Disk Cleanup

Remove any old unused Analytics datasets especially any over 2G in size. The following command can be used to list the datasets.

node:> analytics datasets show

dataset-000 active 745K 2.19M arc.accesses[hit/miss]
dataset-001 active 316K 1.31G arc.l2_accesses[hit/miss]
dataset-002 active 238K 428K arc.l2_size
dataset-003 active 238K 428K arc.size
dataset-004 active 1.05M 2.80M arc.size[component]
dataset-005 active 238K 428K cpu.utilization

Also remove old support bundles or old software updates. It's important that the system disk has at least 20 to 25 percent free space. The following commands can be used to compare the amount of free space to the system disk size. In this case, 289G free / 466G size = 0.62 or 62% free space, which is reasonable. If you have trouble getting enough system disk free space call Oracle Support.

node:> maintenance system disks show
                       profile = mirror
                          root = 4.44G
                           var = 14.3G
                        update = 1.12G
                         stash = 1.38G
                          dump = 131G
                         cores = 29.7M
                       unknown = 15.0G
                          free = 289G


DISK        LABEL      STATE
disk-000    HDD 1      healthy
disk-001    HDD 0      healthy

node:> maintenance hardware select chassis-000 select disk select disk-000 show
                         label = HDD 0
                       present = true
                       faulted = false
                  manufacturer = SEAGATE
                         model = ST95000NSSUN500G
                        serial = 9SP123EQ
                      revision = SF03
                          size = 466G
                          type = data
                           use = system
                        device = c2t0d0
                     interface = SATA
                        locate = false
                       offline = false

Tuesday May 31, 2011

How to reset passwords on your 7000 if you want to start from scratch.

How to reset passwords on your 7000 if you want to start from scratch.[Read More]

This blog is a way for Steve to send out his tips, ideas, links, and general sarcasm. Almost all related to the Oracle 7000, code named ZFSSA, or Amber Road, or Open Storage, or Unified Storage. You are welcome to contact with any comments or questions


« April 2014