frantically waving his hands, AS says....
~loserman~ said:
I usually agree with you but I have to call BS on this one.
...
What you have described in bandwidth is 60Gbits of I/O bandwidth.
You get a B- on your math - it's actually 68 Gbps - each FC is 2 Gbps bidirectional.
Did I claim that I was running at full cross-sectional bandwidth on all ports simultaneously?
No, and it's very common to configure servers with more *connectivity* than they can actually use at any instant.
The Fibre goes to a couple of different SAN arrays.
The GigE goes to several different networks (subnets).
The IB goes to a specialized local network (which is also bridged to GigE and FC).
When the server (it's mostly a disk server) gets a request from a network system, it can service it at full speed (a single disk request is usually disk limited). It can do several at once, all at full speed.
If everything lights up at once, there's a limit - which for these systems seems to be at about 1 GB/sec total throughput in the best case (that's what can move across the system backplane).
This is why 6 to 10 PCI-X slots are typical for a 4U server - not because it can run them all at full bandwidth, but because you want to connect it to many different networks and devices.
If we followed your math, we'd question why an XServe RAID has more than 3 disks... (A disk can do roughly 70 MB/sec, the 2 Gbps FC can do about 200 MB/sec, therefore we'll have an FC bottleneck if we have more than 3 disks.)
The answer, of course, is that we want *connectivity* to more disks so that we can have more MB. It's OK if we hit a limit on MB/sec - as long as we have the MB available.
_________________________
How would you directly connect an XServe to 6 GigE networks, 4 FC SANs, and two IB fabrics?
Y O U __ C A N N O T !!
That's why I've argued for a larger server with more slots, redundant power, and the availability features needed for such use. The XServe is lacking in *connectivity*, and lacking in availability and reliability features.
