Wednesday Sep 08, 2010

Oracle Solaris 10 Update 9: FMA Fixes

It's been a long time since my last blog on FMA - I've since moved into another technology area. Maybe someday I'll garner enough knowledge there to start blogging. In the meantime, with the release of Oracle Solaris 10 Update 9 today, I thought I'd reprise my "favorite FMA fixes" blog entry for this release. So here's my (thankfully short :) list of my favorites:

6627041 Add a PSN nv-pair to the authority portion of the FMRI scheme

This may be less exciting to the end users, but my top favorite. This fix includes the product serial number (PSN) into the FMA fault telemetry. It's a key piece of information to improve the Oracle Auto Service Request (ASR) program. With the PSN, the "Auto" part of ASR is sped up. Hmm...maybe this is exciting for customers - you'll get a faster turn around time on parts servicing.

6502086 DBU errors should be diagnosed as HV defect/fault
6502089 ferg.invalid errors should be diagnosed as a fault

These two CRs provide a level of diagnosis for firmware issues on the sun4v systems. The first covers an error that should not happen unless there's a bug in the hypervisor. The second warrants a little explanation. On CMT systems, FMA telemetry is sourced in the SP - the hardware error is collected in the SP, packaged into an ereport, and provided to Solaris. If the error bits collected can't be mapped to a defined ereport, a "ferg.invalid" ereport is produced. Basically saying "There's been an error, but can't determine what kind". This shouldn't happen outside the lab. If it does, it indicates a logic flaw in the firmware on the SP.

6764337 Needs level 2 FMA compliance for chipset 5100 MCH
6889350 fma fails if DIMM sizes are mixed

A couple of fixes for various Intel configurations. The first enhancing FMA support to include another memory controller chipset. Oracle's CP3250 blades use the 5100 chipset. The second here fixes up a bug - the bug description is self explanatory.

6860401 FMA CPU Topology & Memory Topology needs to support Magny Cours(Multi chip Module)
6812502 Enable Generic-AMD FMA memory topology for Istanbul

These fixes enable CPU and memory diagnosis for the more recent AMD processor offerings. Beyond the typical changes of new model numbers, these processors have a more interesting topology with multiple processing nodes within a single package. Solaris 10 Update 9 understands these chips and creates the correct topology to support FMA diagnosis.


Thursday Jan 14, 2010

FMD Core Dumps and Filling /

snv_132 has a nice fix to avoid a scenario I've seen plenty of times in the lab (and not so much in the field, thankfully :). FMD is a service within SMF and as a service it will be restarted if it crashes. And all things being equal, that's a good thing. FMD also produces a core dump in /var/fm/fmd upon crashing, so we can go figure out what's wrong.

SMF's restarter algorithm includes protection to avoid restarting a service too rapidly. But, the protection parameters are slanted toward a service that starts very rapidly. FMD can take 10s of seconds to start, sometimes longer. If there's a nasty bug in FMD or one of its modules that causes FMD to crash on start, SMF's restarter algorithm doesn't detect this and continually restarts the service. (The same holds true for any service that takes "a while" to start.) And with each restart comes a core file in /var/fm/fmd, which is typically part of the / filesystem. From there, it's only a matter of time before the filesystem fills up.

The restart gap FMD slips through is closed with th integration of:
6219078 svc.startd's algorithm for detecting restart loops should be sensible.


Friday Dec 11, 2009

Rainbow Falls Integration

Several months ago, the Rainbow Falls got press. And today, the processor support integrated into OpenSolaris. While I played a small role in the specifics of the Rainbow Falls project, I'm personally excited about this integration as it's the first SPARC processor to utilize the Platform Independent FMA work I spearheaded roughly one year ago. From an implementation standpoint, Rainbow Falls required few changes to topo enumeration code or diagnosis rules.

Looking forward to the forthcoming products!





« June 2016