T5440 PCI-E Reconfiguration
By pyakutis on Oct 13, 2008
Sun's latest CMT-based server is the four-way Sun SPARC Enterprise T5440. As with the
previous two-way Sun SPARC Enterprise T5140 and T5240 servers, the T5440 is built around the UltraSPARC
T2 Plus processor (an SMP version of the UltraSPARC
T2). The T5440 supports one, two, or four processors in a 4U form factor.
T2 Plus processors, the T5440 provides 256 hardware threads. As with
previous CMT servers the T5440 utilizes PCI-Express (PCI-E) for
Input/Output (IO). With the UltraSPARC T2 and T2 Plus processors, a
PCI-E root complex is brought directly on-chip, reducing latency
between IO devices and memory.
Note that two slots (1 and 6) are physically x16, but electrically x8. Another two
slots (4 and 5) are unavailable if the co-located XAUI slots are used.
The dashed gray line in the figure above shows a x8 PCI-E link that connects between PCI-E switches, but is not necessarily enabled. When all four processors are configured, then PCI-E IO device access to memory or CPU access to a device is through the device's local CPU and the coherence interconnect. (See this diagram.) The x8 crosslinks are not used. However, if the T5440 is configured as a two-way (perhaps to allow for future upgrade to 4-way) then two of the switches will not have a local root complex. In that case, two of the crosslinks will be enabled to preserve full connectivity to IO. The following diagram shows the supported two-way T5440 configuration.
With the supported two-way configuration, the on-board LSI1068E SAS controller for the internal disks, as well as the Neptune ASIC used for network connectivity retain direct connections to local root complexes (CPU0 and CPU1 respectively). However, devices connected through slots 2,3,6,7 obtain connectivity to a root complex through an extra PCI-E switch hop. This will add latency to the I/O path for those devices. (Full path of remote memory access.)
Note that changes in the PCI-E configuration may change the Solaris path names for devices. For example, if a customer starts with the two-way configuration shown above and later upgrades to a four-way configuration, devices connected using slots 2, 3, 6, and 7, will have different Solaris path names. The Sun SPARC Enterprise T5440 Server Product Notes provide more detail concerning PCI-E reconfiguration including supported configurations, reconfiguration procedures, and use of an available reconfiguration script.
Reconfiguration can be useful in the event of a processor failure, as well. If a processor failed, then access to that processor's PCI-E IO devices would be unavailable. Probably the best solution would be to replace the CPU, but if one is not immediately available, then by using reconfiguration, access to those I/O devices can be restored to the system. Reconfiguration also allows a customer to start with a single CPU and memory, and later add processors, going to two-way or four-way as needed. T5440 PCI-E reconfiguration provides flexibility and full access to I/O.