So let me see if i have this straight. From what you are reading and seeing is the following correct?
IP over TB:
- Auto negotiation
- Only Mavericks Peer to Peer connections
- Uses a modified OSI stack, i.e. incomplete TCP ??
- Does not require (or support?) switching (Layer 2)
No
The underlying base technology -- Thunderbold -- is able to use different topologies (daisy-chaining, star, tree). It's a host based technology (like USB) but it's also possible to build data paths between different hosts (not only "one host, many peripheral devices").
IP over Thunderbolt is such a 'host to host' setup. If you connect two Macs directly then this is a direct TB bus able to utilize the entire bandwidth of one TB channel. If there are other TB devices in endpoint mode in between (daisy chaining) it is still one TB bus but the maximum speed might depend on how the controller assigns datapaths to the available channels:
http://www.tomshardware.com/reviews/thunderbolt-performance-z77a-gd80,3205-4.html
Two remarks:
- if I understand correctly with TB2 we don't have 2 independent channels but one delivering twice the bandwidth (therefore compatibility with TB1 cables)
- unlike SCSI or FW the topology on a TB bus is different because it depends on where the upstream host controller is plugged in (the TB ports in devices don't act in pass-through mode but there's an endpoint controller inside each device which connects to the host controller upstream and passes TB packets down the cable on the other port if they should reach devices behind). So if you have a couple of endpoint devices on the bus (storage devices, a thunderbolt display) and connect another mac to the end of this chain it won't lead to chaos.
IP over Thunderbolt always utilizes point-to-point-connections at the physical layer: If you have a Mac with 2 TB ports and connect to each of them another Mac then these are two independent TB buses.
To exchange data with these buses (or machines since on any bus can only be one other Mac active) Mavericks establishes a bridge device. If you assign only one TB port/bus to this bridge device then network traffic will only flow between those two Macs unless you enable packet forwarding (routing) on the TCP/IP layer (sysctl net.inet.ip.forwarding=1) which will lead to the OS routing packets between different IP enabled interfaces if routing is set up properly.
This applies also to the scenario where you have two TB ports but decide to let them be assigned to two different bridge devices instead of one. In this case there won't be any packet exchange on this mac unless you enable packet forwarding in the OS.
Now my assumptions:
If you have more than one TB port
and assign two of them to the same bridge device than something different happens: Not visible from the network layer of the OS a lower layer (driver) acts as a multiport bridge between the two separate TB buses (maybe on layer 2 if it's a real bridge forwarding all sorts of packets maybe layer 3 based and restricted to IP traffic). Only packets that target the local host will be available at the bridge device and appear in the network stack of OS X, all other traffic will be bridged between the 2 otherwise independent TB busses.
If the new MacPro will be available then we can have a deeper look whether this bridging works like a hub (forwarding all packets to all different TB buses) or like a switch (checking the packet's target MAC address and forwarding it only on the TB bus where this device resides). I would believe the whole thing works like a switch because packet flooding all buses at these speeds is sort of stupid.
The bridge interface advertises to the OS that it is capable of doing large segment offloading and offload checksumming. In traditional Ethernet environments this is done by more expensive NICs to free the CPU from calculating checksums for rather small packets (based on the MTU settings we inherited from the seventies of the last century). When the NIC provides these capabilties the OS can send packets up to 64 KB in size to the NIC device and a special engine on the NIC itself starts to reassemble the packets to the MTU value (1500 byte historically). The TB bridge device advertises this capability ("options=63<RXCSUM,TXCSUM,TSO4,TSO6>") but won't do any offloading stuff (which had to be done entirely in software because there is no special NIC hardware to assist) but simply sends these large packets on the TB buses (segmented in smaller TB packtes of course) connected to the bridge device. So even if the device announces that it is capable of a MTU of only 1500 bytes in fact packets of as nearly large as 64 KB will be transmitted over the wire (VMTU --> virtual MTU from the point of view of the TCP/IP stack of the OS).
Offloading will only happen if you also assing a traditional Ethernet device to one of the bridge devices since in this case the lower layers (below/hidden from the TCP/IP stack of OS X) have to reassemble the packets arriving at the bridge device with large MTU values so that it fits in the MTU of the LAN/WLAN device that is also part of the bridge device.
Apple chose to use the classical Ethernet frame format for the IP over Thunderbolt stuff. But I'm not really sure if it's really "Ethernet over TB" as some people claim (since then the bridge device should be capable to handle all sorts of different protocols and might even have to change the size of Ethernet frames which isn't possible at layer 2 to my knowledge). I would believe it's really "IP over thunderbolt" since packet fragmentation can be done without any problems on layer 3.
I have a Mac with 2 TB ports. Has anyone tried daisy chaining two other macs to the two ports and tried to route traffic between the two end hosts?
From Mac A to Mac C below:
{Mac A}--------{Mac B}--------{Mac C}
I am assuming this would be no problem, but the topology is extremely limiting.
In this case you have 2 independent TB buses: A-B and B-C (and daisy chaining is
not involved because this would mean that you have devices in endpoint mode in between). If you assign both ports to one bridge device on Mac B then this Mac will act as some sort of a multi-port bridge between both buses (maybe on layer 2 forwarding all sorts of Ethernet like frames maybe just on layer 3 only handling IP protocol stuff). If this assumption is true then you wouldn't have the chance to sniff any packets on Host B that will be sent from A to C or vice versa.
You spoke about "routing". This would be the case if you assing both TB ports on Mac B to different bridge devices and enable packet forwarding in between. Then every packet betwen A and C has to pass the upper layers of OS X' network stack (routing between interfaces).
BTW: A totally different scenario would also be possible. Today we have only TB endpoint devices with 2 ports. Therefore the only TB topology in the wild is 'daisy chaining' (and the new MacPro won't change this because it is a device with 6 TB controllers in host mode each defining the endpoint of one independant TB chain/topology). TB could also use different topologies like tree or star. In this case we would see routing/switching of raw packets on the TB layer (neither 'IP packets' nor 'Ethernet frames' -- the whole thing is something different happening below).
I have no idea how "IP over Thunderbolt" would deal with such a situation where on one TB bus connected to the TB port of a Mac is more than one other Mac present.
