The way I read it is: the flash chip(s) expect a PCIe bus (not being Apple-specific), but ANS merely pretends to offer one. Apple's SSD controller then uses the other ANS for the other side of the communication.
I'm not sure. The language isn't clear. For instance, the word "fronts" in the following....
"M1 contains a coprocessor, called ANS, that
fronts the actual PCIe bus to the NVMe device."
...would typically means that it's CPU->ANS ->PCIe -> NVMe
Are you saying that, when you run ioreg, it unambiguously indicates the path is CPU -> ANS -> NVMe?
And have you (unlike me) been able to find any independent documentation of this on the internet? Usually one can find multiple hits on pretty much everything Apple, so this seems curious.
Not a lot.
It doesn't (see above). The Mac Pro uses a switch to reuse lanes.
Yeah, I saw that tweet previously. The Intel MP does the same thing: Like the AS MP, it doesn't have enough PCIe I/O to support all its PCIe connections at full bandwidth. The difference with the AS MP is that the discrepancy between slots and bandwidth is much larger.
Hector says the M2 Ultra has 32 PCIe lanes:
I'm not sure how many PCIe lanes the M2 Ultra's ports take up. If it's as follows, and if no PCIe lanes are needed for storage...
6 TB4 ports x 4 lanes/port = 24 lanes
10 Gbs Ethernet = 1 lane
HDMI 2.1 = 3 lanes?
UHS-II SXDC + headphone + 2 x 5 Gbs USB = 1 lane?
....then the Ultra can only make use of ~ 30 lanes, which would mean that the MP does offer more I/O, but only modestly so. Feel free to update the above with correct lane values if you have them.
It seems the real value of the MP is (a) the general ability to install PCIe cards without needing a separate box; and (b) the specific ability to aggregate I/O into a single fast connection. E.g., you could use up half of your PCIe lanes (16/32) on one of these, which would provide a (claimed) 208 Gbps data transfer rate, equivalent to more than 5 TB4 connections:
[If the claim is true, than that would provide a significant boost in I/O over the Ultra, since it would take 20 PCIe lanes to support 5 TB4 connections at full bandwidth; it sounds like when you have 4 PCIe lanes/TB4 port, some of the PCIe bandwidth goes unused. By aggregating those into a single higher-banwidth connection, you avoid that loss.]
www.cdrinfo.com
"HighPoint’s Dual-Width E1.S and M.2 NVMe AICs are the perfect High-performance storage solution for Apple’s new powerhouse. Designed for high-stress media, industrial and AI applications, SSD7749 series NVMe RAID AIC host bus adapters, are capable of delivering up to 28GB/s of sustained transfer throughput via a single PCIe Gen4 x16 slot, and double-this with a Cross-Sync configuration!"