Monday Oct 13, 2008

T5440 PCI-E Reconfiguration

Sun's latest CMT-based server is the four-way Sun SPARC Enterprise T5440. As with the previous two-way Sun SPARC Enterprise T5140 and T5240 servers, the T5440 is built around the UltraSPARC T2 Plus processor (an SMP version of the UltraSPARC T2). The T5440 supports one, two, or four processors in a 4U form factor.  With four T2 Plus processors, the T5440 provides 256 hardware threads.  As with previous CMT servers the T5440 utilizes PCI-Express (PCI-E) for Input/Output (IO). With the UltraSPARC  T2 and T2 Plus processors, a PCI-E root complex is brought directly on-chip, reducing latency between IO devices and memory.
Each on-chip PCI-E x8 root complex connects directly to a PLX PEX8548 PCI-E switch. In turn, each switch connects to on-board IO as well as two PCI-E slots. With four switches, there are a total of eight PCI-E slots. The following diagram shows the the PCI-E topology of the fully configured four-way T5440.

Note that two slots (1 and 6) are physically x16, but electrically x8. Another two slots (4 and 5) are unavailable if the co-located XAUI slots are used.

The dashed gray line in the figure above shows a x8 PCI-E link that connects between PCI-E switches, but is not necessarily enabled. When all four processors are configured, then PCI-E IO device access to memory or CPU access to a device is through the device's local CPU and the coherence interconnect. (See this diagram.) The x8 crosslinks are not used. However, if the T5440 is configured as a two-way (perhaps to allow for future upgrade to 4-way) then two of the switches will not have a local root complex. In that case, two of the crosslinks will be enabled to preserve full connectivity to IO. The following diagram shows the supported two-way T5440 configuration.

With the supported two-way configuration, the on-board LSI1068E SAS controller for the internal disks, as well as the Neptune ASIC used for network connectivity retain direct connections to local root complexes (CPU0 and CPU1 respectively). However, devices connected through slots 2,3,6,7 obtain connectivity to a root complex through an extra PCI-E switch hop. This will add latency to the I/O path for those devices. (Full path of remote memory access.)

Note that changes in the PCI-E configuration may change the Solaris path names for devices. For example, if a customer starts with the two-way configuration shown above and later upgrades to a four-way configuration, devices connected using slots 2, 3, 6, and 7, will have different Solaris path names. The Sun SPARC Enterprise T5440 Server Product Notes provide more detail concerning PCI-E reconfiguration including supported configurations, reconfiguration procedures, and use of an available reconfiguration script.

Reconfiguration can be useful in the event of a processor failure, as well. If a processor failed, then access to that processor's PCI-E IO devices would be unavailable. Probably the best solution would be to replace the CPU, but if one is not immediately available, then by using reconfiguration, access to those I/O devices can be restored to the system. Reconfiguration also allows a customer to start with a single CPU and memory, and later add processors, going to two-way or four-way as needed. T5440  PCI-E reconfiguration provides flexibility and full access to I/O.


About

pyakutis

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today