The Retina iMac was introduced in 2014, and featured Thunderbolt 2 (max data rate 20 Gbps). By comparison, the 27" 5k Dell UP2715, also introduced in 2014, was driven with dual DP1.2 cables, and DP1.2 has a max data rate of 17.28 Gbps. [Sometimes you see a specification of 21.6 Gbps, but that includes the overhead.]
So clearly it was possible for a 5k 60 Hz display to be driven by the tech available when the Retina iMac was developed. Indeed, DP1.2 became available in 2010.
Further, unless there's some wrinkle in TB2 that limits its video-specific bandwidth, it additionally could have been driven by the specific TB tech that Apple used at the time (TB2 was introduced by Apple a year earlier, in 2013).