IB network design leaf and core switch trade-off

Current IB switch based on Mellanox IS III 24 port 4x chipset support upto 288 in 3 stages and 3456 in 5 stages.
Recently Sun has build in leaf switch in the Blade 6048 rack.
If one use 3stages IB switches with sb6048 build in leaf switch one will create 5 stages network, and if one use 5 stages IB switch then we will create 7 stage network.
TACC support more than 3456 nodes so it need 2 3456 switches and it use IB-NEM to create 7 stage CLOS network
For smaller network (<=288) one could use many 24 port switches as core swicthes to build a 3 stages IB network.
  • 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require 3x24 port core switches
  • 8 chassises of 12 blades: 2 sb6048 = 96 blade require 4x24 port core switches
  • 12 chassises of 12 blades: 3 sb6048 = 144 blades require 6 x24 port core switches
  • 24 chassises of 12 blades: 6 sb6048 = 288 blades require 12 x24 port core switches
5 stages network using NEM IB leaf switch
  • 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
  • 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
  • 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
  • 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch
3 stages network using IB PCI-E EM :
  • 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
  • 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
  • 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
  • 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch
For much larger network (>288) one could use many 288 port switches as core switches to build a 5 stages IB network. one can also use the dual port IB EM modules connect to core switches , for <=288 of 3 stages network and <=3465 for 5 stages networks.
Of course one use many core switches to construct larger IB network.
The same argument can apply to rack with rack server:
there are design tradeoff with big IB switch and has small leaf switches in each rack.
for examples if we use 36 1U server as building block and 3 24 port leaf switches in a rack , then there are two way to build IB network
  • use 3 stages core switches (<=288) to build a 5 stages network
  • use many 24 port switches as core switch for 3 stages network
The main difference between 3 stages and 5 stages IB network is the two switch hub difference: ~2x200 nanosecond difference for 24 port switch chipset.
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

hstsao

Search

Top Tags
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today