So there appears to be three different bottlenecks in play here. 1) The actual cable and chipset in the cable, and each port, which can handle 20Gbps, 2) The Thunderbolt 2 bus itself which also tops out at 20Gbpss, and 3) the DisplayPort 1.2 connection to the GPU which can pump out enough bandwidth to do 4k at 60hz (by pretending it's two displays using MST).
Soooooo, with this theory the TB2 connection using Port A going to the drive array would have the entire TB2 bus to use, and the Display Port 1.2 connection to the display using Port B would also have the full bandwidth from the GPU available to it.
Each port does not provide 20 Gbps. There is only one Thunderbolt bus in the iMac with 2 ports (historically and I am pretty sure it's the same this time).
It is 20 Gbps in total for the bus. The display port signal will consume its own chunk out of that 20 Gbps and whatever is left will be the bandwidth for your external hard drive enclosure. Your sequential reads and writes will be affected a bit.