Friday Aug 01, 2008

The FMA Triad: Topology, Telemetry & Diagnosis Rules - Part 4

Over the course of the last several months, I've described how topology, telemetry, and diagnosis rules work together within the Fault Management Architecture (FMA). Something I've dubbed the FMA Triad. It's high time I finished off this little mini-series with an example of what happens when the members of the Triad don't play nicely together.

Part 4 - When Things Go Wrong

This is a real world example. I briefly mentioned at the tail end of Part 3 that the diagnosis rules often use relative FMRIs, but the telemetry and the topology must use fully qualified FMRIs. When there's a disconnect, the diagnosis engine cannot determine the system resource and is unable to understand the incoming telemetry.

Customers upgraded their T2000 systems to Solaris 10 Update 4 and when starting the OS were greeted with this:

SUNW-MSG-ID: SUNOS-8000-1L, TYPE: Defect, VER: 1, SEVERITY: Minor
EVENT-TIME: Feb 08 15:30:39 CST 2008
SOURCE: eft, REV: 1.16
EVENT-ID: 1be16b2d-158e-e73b-f097-b744c2eb8cd3
DESC: The EFT Diagnosis Engine encountered telemetry for which it is unable to produce
a diagnosis.  Refer to for more information.
AUTO-RESPONSE: Error reports from the component will be logged for examination by Sun.
IMPACT: Automated diagnosis and response for these events will not occur.
REC-ACTION: Run pkgchk -n SUNWfmd to ensure that fault management software is installed
properly. Contact Sun for support.
Ouch. Not good. Not only is there an apparent hardware problem, FMA can't make heads or tails of it. First thing to do is find out what events led to this "diagnosis". We can use fmdump for that:

# fmdump -e -u 1be16b2d-158e-e73b-f097-b744c2eb8cd3
TIME                 CLASS
Feb 08 15:27:30.4960
Ok. After checking the event registry, "lup" is a link up error report. Normally, the diagnosis engines ignore these error - and on boot we flat out expect them (the links have to come up). In fact, I tipped my cards in Part 3 by showing you the rules for a "lup" error. Looking deeper at the telemetry:

# fmdump -eV -u 1be16b2d-158e-e73b-f097-b744c2eb8cd3
TIME                           CLASS
Feb 08 2008 15:27:30.496065920
nvlist version: 0
        class =
        ena = 0xa3e9ab80002
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = hc
                hc-root =
                hc-list-sz = 3
                hc-list = (array of embedded nvlists)
                (start hc-list[0])
                nvlist version: 0
                        hc-name = ioboard
                        hc-id = 0
                (end hc-list[0])
                (start hc-list[1])
                nvlist version: 0
                        hc-name = hostbridge
                        hc-id = 0
                (end hc-list[1])
                (start hc-list[2])
                nvlist version: 0
                        hc-name = pciexrc
                        hc-id = 0
                (end hc-list[2])

        (end detector)

        primary = 1
        tlu-oeele = 0xffffff
        tlu-oeie = 0xffffff00ffffff
        tlu-oeis = 0x100
        tlu-oeess = 0x0
        __ttl = 0x1
        __tod = 0x47ace562 0x1d915d80
The telemetry is describing an FMRI of hc:///ioboard=0/hostbridge=0/pciexrc=0, but the diagnosis rules aren't understanding that. But, the rules are only looking for errors against hostbridge/pciexrc. So the rules themselves are fine. The problem must be that this FMRI can't be located in the topology. Turning to fmtopo:

# /usr/lib/fm/fmd/fmtopo
Eureka! There is in fact a disconnect between the telemetry and the topology. This explains the undiagnosable errors from FMA.

Now, to the question of what changed to cause this problem? On T2000 class systems, telemetry for root complex errors (like the "lup" error) is generated in the Service Processor. Topology is constructed by enumerators in Solaris. Since the change made to the system was an upgrade of Solaris, something has gone wrong in Solaris 10 Update 4.

The mechanisms for generating a topology changed significantly between S10U3 and S10U4. In S10U3, there were .topo files, and the one for Netra,T2000 described an ioboard/hostbridge/pciexrc arrangement. When the newer XML map mechanism came with S10U4 (the one I described in Part 1 of this series), a platform specific topology map for Netra,T2000 was overlooked. As Part 1 details, when there is no platform specific topology map, FMD reverts to the architecture specific topology map - sun4v in this case. The sun4v XML map in S10U4 describes a motherboard/hostbridge/pciexrc arrangement.

The solution was to provide a platform specific topo map for the Netra,T2000 system. If memory serves, a similar change for the Netra,CP3060 was also needed.

I hope you found this to be a good example that ties this series up. If you've enjoyed this half as much as I have, then I've enjoyed it twice as much as you. :)


Tuesday Jul 29, 2008

Putback: Platform Independent FMA for sun4v

As predicted in my last entry about sun4v platform independent FMA, I am going to yelp about the putback:

Event: putback-to Comment: FWARC 2008/300 Sun4v FMA Platform Independent FMA Topology Enumeration PSARC 2008/392 FMA new canonical hc names for sun4v_pi enumerator PSARC 2008/440 sun4v Platform Independent Topology Enumerator - libmdesc extensions 6628827 Need platform independent topo enumeration for sun4v platforms Files: update: usr/src/lib/fm/libmdesc/ update: usr/src/lib/fm/libmdesc/common/mapfile-vers update: usr/src/lib/fm/topo/libtopo/common/hc.c update: usr/src/lib/fm/topo/libtopo/common/topo_hc.h update: usr/src/lib/fm/topo/maps/sun4v/sun4v-hc-topology.xml update: usr/src/lib/fm/topo/modules/sun4v/Makefile update: usr/src/lib/fm/topo/modules/sun4v/pcibus/Makefile update: usr/src/lib/fm/topo/modules/sun4v/pcibus/pci_sun4v.c update: usr/src/pkgdefs/SUNWfmd/prototype_sparc update: usr/src/uts/common/sys/mdesc.h create: usr/src/common/mdesc/mdesc_getproparcs.c create: usr/src/common/mdesc/mdesc_walkdag.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/Makefile create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_cpu.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_defer.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_generic.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_impl.h create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_ldom.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_niu.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_pciexrc.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_subr.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_top.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/pi_walker.c create: usr/src/lib/fm/topo/modules/sun4v/sun4vpi/sun4vpi.c Examined files: 24 Contents Summary: 14 create 10 update

The code went into snv_96 yesterday so it should hit OpenSolaris in a day or so.


Wednesday Jul 02, 2008

Platform Independent FMA for sun4v: July 2008 update

It's been a little over a half a year since I first mentioned the "Platform Independent FMA for sun4v" project (aka FMA PI) in this blog. The very keen observers may have noticed PSARC 2008/392 was filed and approved. The material in the case itself is, well, unexciting. It's a short list of new hardware component (hc) scheme canonical names - the list of hc names that FMA recognizes as valid. But its the work behind it that's been very cool. So figured it was time for an update.

There's two active sub-projects at the moment. First is the development of a sun4v generic topology enumerator. This project is quite far along - the code is in test, and should be integrated into OpenSolaris in the next 30 days. PSARC 2008/392 is the first visible result of the project. The PSARC case really doesn't give much insight into how the enumerator is working, so a short description of how it works is in order.

On sun4v systems, the firmware in the SP provides a Physical Resource Inventory (PRI). The contents of this structure, as its name suggests, details all of the physical attributes of the platform. It is primarily for use with Logical Domains (LDOMs), but some portions of it are consumed by other subsystems, FMA being one of them. The PRI already had\\s the capability to describe relationships between components, much like a topology map file was doing in Solaris. So with FMA PI, we defined a set of new properties for certain PRI nodes that our new Solaris-side enumerator picks up to construct the FMA topology. End result - sun4v platforms control their topology from the SP firmware. We can take a PRI node that looks something like this:

node component node_0x3dac { back -> node_0x38aa; type = "dimm"; fru = 0x1; label = "J585"; topo-hc-name = "dimm"; part_number = "511-1161-01 Rev 01"; serial_number = "00CE0108042425E8B3"; dash_number = "00"; rev_number = "00"; sdram_manuf_jedec_id = "80CE"; amb_vid = "111D"; amb_rid = "21"; amb_did = "0482"; id = 0x0; nac = "MB/CPU0/CMP0/BR0/CH0/D0"; back -> node_0x3da4; }

and turn it into an FMA topology entry that looks like this:

hc://:product-id=SUNW,FOO:server-id=sca-foo-1/chassis=0/motherboard=0/cpuboa rd=0/chip=0/memory-controller=0/branch=0/dram-channel=0/dimm=0 ASRU: - FRU: hc://:product-id=SUNW,FOO:server-id=sca-foo-1:serial=00CE010804 2425E8B3:part=511-1161-01 Rev 0100:revision=00/chassis=0/motherboard=0/cpuboard= 0/chip=0/memory-controller=0/branch=0/dram-channel=0/dimm=0 Label: MB/CPU0/CMP0/BR0/CH0/D0

The only thing I've not reflected above is the hierarchy built into the PRI. But the fwd and back nodes give you the notion. Once the enumerator is putback to Solaris, sun4v platforms don't have to write enumerators anymore. The topology is encoded into the FW, and the FMA PI enumerator transforms it into a usable topology.

The other active project is for SPARC generic CPU and memory diagnosis. This project is still in the earlier stages of development. Processor-agnostic ereports have been defined and are being fine tuned. To follow is the diagnosis rules to consume those ereports. An analogy to the project is the MCA work done on x86 - sun4v FMA PI aims to provide a base level of FMA support for any sun4v platform. But with sun4v, as there's more that can be done with the SP firmware, we're targeting a stronger base feature set. Since the project is still early on, I'll leave it at that for now.

If you've read any of my "FMA Triad" posts, you may notice the pattern to the projects: topology, telemetry, and diagnosis rules. I'll endeavor to give more updates on the FMA PI projects as the rest of 2008 progresses. At the very least, you'll likely hear from me when the OpenSolaris putback happens.





« June 2016