Wednesday Sep 08, 2010

Oracle Solaris 10 Update 9: FMA Fixes

It's been a long time since my last blog on FMA - I've since moved into another technology area. Maybe someday I'll garner enough knowledge there to start blogging. In the meantime, with the release of Oracle Solaris 10 Update 9 today, I thought I'd reprise my "favorite FMA fixes" blog entry for this release. So here's my (thankfully short :) list of my favorites:

6627041 Add a PSN nv-pair to the authority portion of the FMRI scheme

This may be less exciting to the end users, but my top favorite. This fix includes the product serial number (PSN) into the FMA fault telemetry. It's a key piece of information to improve the Oracle Auto Service Request (ASR) program. With the PSN, the "Auto" part of ASR is sped up. Hmm...maybe this is exciting for customers - you'll get a faster turn around time on parts servicing.

6502086 DBU errors should be diagnosed as HV defect/fault
6502089 ferg.invalid errors should be diagnosed as a fault

These two CRs provide a level of diagnosis for firmware issues on the sun4v systems. The first covers an error that should not happen unless there's a bug in the hypervisor. The second warrants a little explanation. On CMT systems, FMA telemetry is sourced in the SP - the hardware error is collected in the SP, packaged into an ereport, and provided to Solaris. If the error bits collected can't be mapped to a defined ereport, a "ferg.invalid" ereport is produced. Basically saying "There's been an error, but can't determine what kind". This shouldn't happen outside the lab. If it does, it indicates a logic flaw in the firmware on the SP.

6764337 Needs level 2 FMA compliance for chipset 5100 MCH
6889350 fma fails if DIMM sizes are mixed

A couple of fixes for various Intel configurations. The first enhancing FMA support to include another memory controller chipset. Oracle's CP3250 blades use the 5100 chipset. The second here fixes up a bug - the bug description is self explanatory.

6860401 FMA CPU Topology & Memory Topology needs to support Magny Cours(Multi chip Module)
6812502 Enable Generic-AMD FMA memory topology for Istanbul

These fixes enable CPU and memory diagnosis for the more recent AMD processor offerings. Beyond the typical changes of new model numbers, these processors have a more interesting topology with multiple processing nodes within a single package. Solaris 10 Update 9 understands these chips and creates the correct topology to support FMA diagnosis.


Tuesday Nov 03, 2009

Putback: x86gentopo Project

A few months ago, I talked about the x86gentopo project - a reusable baseboard enumeration approach for x86 systems. It's the same spirit that drove the platform independent sun4v FMA work.

Yesterday evening, the project integrated into OpenSolaris. If you missed the flag day message, here it is again.

If you don't use x86 systems, or don't care about x86 FMA, you can hit delete now. There is a new x86 generic FMA topology enumerator available with the putback of: PSARC/2009/490 x86 Generic FMA Topology Enumerator 6785310 Implement SMBIOS contained elements/handles 6841286 Need x86 generic FMA topo enumerator 6853537 x86gentopo needs OEM-Specific SMBIOS structures 6865771 Topology relationships should be derived from contained handles & elements of SMBIOS 6865814 Chip enumerator should derive serials & labels using libsmbios, if SMBIOS is FM aware 6865845 /dev/fm should export the Initial APICID, SMBIOS based ID/instance to the chip enumerator 6866456 Generic Topology FMRI ereport The new x86 generic enumerator creates physical topology, as well as identity information (serial number, part number, etc...), for i86pc class systems which contain a compatible SMBIOS. The X64 Platform Resource Management Specification (PRMS-1) describes what a compliant SMBIOS is (currently in uncirculated draft form). To correctly diagnose faults the cpu and memory ereport generators have been modified to report x86 generic topology, when a compliant SMBIOS is found. If a compliant platform SMBIOS is not found, the x86 generic enumerator and x86 generic ereport generators will revert to existing (legacy) enumeration and ereport generation. If a platform does contain a compliant SMBIOS and wishes to force legacy enumeration, the kernel tunable variable x86gentopo_legacy can be set in /etc/system: set x86gentopo_legacy = 1 To report bugs against the x86 generic topology enumerator and/or the cpu/mem ereport generators please use the following product/cat/subcat: solaris/fma/other : x86 generic enumerator solaris/fma/mem : memory ereport generator solaris/fma/cpu : cpu ereport generator To report bugs against the SMBIOS structure use please use the following product/cat/subcat: solaris/library/libsmbios For more information please visit the OpenSolaris x86gentopo project page: Thank you, The x86gentopo team

Now to get that PRMS finalized and made public...


Thursday Oct 22, 2009

Solaris 10 Update 8 FMA Fixes

Solaris 10 Update 8 is now posted and available for download. And there's been plenty of bug fix work for FMA. Beyond lots of memory leak and core dump fixes, here's my favorite fixes.

6743295 fault.memory.dimm is overloaded
6758561 KA pages for fault.memory.dimm\* are needlessly different

These two fixes provide some cleanup for DIMM faults. CR 6743295 explains the more tangible benefit, IMHO. In addition to getting the information a DIMM is declared faulty, there is now better separation via the fault and knowledge article explaining why the DIMM is deemed bad.

6394503 fmdump should show contents of rotated logs without specifying them explicitly
6535637 Add Severity level to payload of list.suspects event

I lump these together as administrative improvements. The first expanding fmdump to show information from historical logs and not just the current log. FMA error and fault logs are rotated periodically, and the new behavior is to display data from all logs.

6618751 Include memboard in T5440 FBR/FBU diagnosis

Beyond fixing a nasty core dump in the diagnosis flow, this fix also improves the diagnosis. This was a gap from when T5440 first shipped. On configurations where DIMMs on memory boards are in the mix, the memory board itself is part of the FB-DIMM channel. That component is now included in the diagnosis of channel errors.

6747341 Add FMA to hermon driver
6656720 Initial hxge driver

Add two more FMA hardened drivers to Solaris 10!


The impetus for this change is the higher core counts coming in the x86 world. With newer chips from both AMD and Intel, the previously defined maximums would be insufficient to fault manage all the cores and strands. Not anymore :)

6818561 FMA topology fails on Sun Blade T6300

This was just a flat out embarrassment. Topology completely missing from the T6300 blades, rendering FMA largely ineffective (errors still logged, but diagnosis crippled). Glad this one is fixed.





« September 2016