You can't connect any other display to the same Thunderbolt controller if the first display is an LG UltraFine 5K. Redo your tests without the LG UltraFine 5K connected.
The Apple Studio Display requires HBR2+DSC for 5K60 or it requires two DisplayPort HBR2 connections like the LG UltraFine 5K. It does not support HBR3 (but neither does the Alpine Ridge).
The 1080 TI supports DisplayPort 1.4 HBR3, no DSC.
Extract the EDID of the Apple Studio Display using CRU. Does CRU show two connections? Post the results here. If you can't get dual DisplayPort connection to work, then maybe you need to remove the 5K timing from the EDID? But before you do that, see if you can get 5K60 using 6bpc.
If you had two GC ALPINE RIDGE, maybe you could connect the LG UltraFine 5K to one and the Apple Studio Display to the other. You would need four DisplayPort connections. I don't think the 1080TI has that many, but maybe you could use two DisplayPort 1.4 MST hubs to double the DisplayPort connections. You would be limited to 5K 60Hz 8bpc since the 1080TI doesn't support DSC. Otherwise, a DisplayPort 1.4 MST hub can decompress DSC to produce the uncompressed HBR2 outputs.
Which version of the LG UltraFine 5K do you have? The 27MD5KL can use a non-Thunderbolt or 20 Gbps connection which only uses up one DisplayPort connection. It allows 4K60. It may allow 5K39 8bpc.
I don't know yet what the max pixel clock of the Apple Studio display is using a non-Thunderbolt or 20 Gbps connection. It might allow more than 39Hz for 5K 8bpc (up to 47 Hz?).