So this is not for the faint heart. I am going to give you a roller coaster ride going at 40 Gb/s. ;) Used some shark teeth and cut the InfiniBand packets apart to show you some screenshots here.
Just kidding, dont worry. This post is about how to capture InfiniBand packets and then view them in latest Wireshark. Mellanox has generously released a nice and powerful utility called ibdump. You can visit their website and get a copy till supplies last. So hurry up !
I suggest saving the dums on tmpfs and transfer elsewhere later. Use the --mem-mode for precise and fast captures.
MAD packet with NodeDescription
IPv4 ping (ICMP) on IPoIB interface. You should notice that we are using connected RC mode for underlying InfiniBand. Also notice the timestamps of ICMP and IB packets.
I am still working on these for various analysis. So, please return here for more screenshots later. By the way, the wireshark interface with InfiniBand captures is fairly new so you may see a few erroneous reports especially when you turn on payload parsing. But its still a lot of fun. :)