Studio display natural resolution?

[joevt]

I can't find the actual model # right now, but it's the last 17" MBP w/ Touch Bar that Apple made.

screenshot of Apple menu -> About This Mac
I think the 5500M should support DSC but I'm not 100% sure.

Our IT department is unlikely to upgrade to Monterey anytime in the next year, what would you suggest I set the display to, to get the most out of it?

You could use an external USB or Thunderbolt NVMe, install macOS and whatever else you like. Then use Migration Assistant to get all the apps and docs from the internal install.

@joevt — Any ideas?

Need output from AGDCDiagnose on the Intel Mac with AMD 5500 GPU to see if the display supports DSC and what the max DisplayPort link rate is. For a tiled display, it can get the EDID for both tiles.

/System/Library/Extensions/AppleGraphicsControl.kext/Contents/MacOS/AGDCDiagnose -a > AGDCDiagnose_a.txt 2>&1

On the M1 Mac, can you run the script at https://gist.github.com/joevt/e862b0088ef58b9144877d01401bcee8
This will show if the M1 Mac is using a tiled mode or a single tile mode.

Also on the Intel Mac, try a USB-C cable instead of a Thunderbolt cable and redo the AGDCDiagnose command. A USB-C cable will limit the connection to 20Gbps which is sufficient for 5K60 only if DSC is supported. The point of the USB-C connection is to eliminate the ability to have two DisplayPort connections over Thunderbolt which will remove the ability to do tiled mode. But in the case of a 20 Gbps connection, it is still Thunderbolt. For a pure DisplayPort connection, a Belkin Charge and Sync cable is best.

Take a screenshot of the Display Information tab for the display in SwitchResX for each connection type to see if the frequency range changes in each case.

I haven't installed Monterey 12.3 yet so I don't know what new override files exist for the Studio Display.
/System/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610
There's probably a new .mtdd file to describe the tiled display (named something like DisplayProductID-ae3a.mtdd ). It's only needed if you need tiled mode to get 5K60. It may be sufficient to copy that from Monterey 12.3 to earlier macOS versions. Catalina would be easy since you can simply make the root volume writable temporarily. The process is more difficult for Big Sur and Monterey. I don't know if the .mtdd file can work from the /Library/.../DisplayVendorID-610 folder instead of /System/Library/.../DisplayVendorID-610 folder.

The AllRez command ( #177 ) can list all the modes (including pixel format info such as DSC, or chroma sub sampling - this info depends on the GPU drivers - older drivers don't provide pixel format info). It also shows DisplayPort and EDID info like AGDCDiagnose (but AGDCDiagnose may be better at getting EDID). And it includes frequency range info. It currently works best on Intel Macs but produces some output on an M1 Mac which I can use to produce better output for M1 Macs.

Catalina might be best for testing DSC support (if AGDCDiagnose says DSC is supported and it won't work in Big Sur or Monterey).

For Intel Macs, I have a fork of the Lilu and WhateverGreen kexts which can do the patch that is enabled by the -cdfon boot-arg for all versions of macOS. The patch stops CoreDisplay in WindowServer from filtering out some display modes. It might be interesting to see what modes those are (use AllRez to list before and after the patch). The usual way of using Lilu and WhateverGreen is to use OpenCore to inject them. Installing them to /Library/Extensions might work, or they might not load early enough (before WindowServer starts). In that case, the kill command can restart the WindowServer process which might then allow the patch to be made. If you use any other Lilu based kexts, then you may need to recompile them using my kext.
https://www.insanelymac.com/forum/t...ocess-patcher/?do=findComment&comment=2778421
https://github.com/joevt/Lilu
https://github.com/joevt/WhateverGreen
OCLP is a method to automatically install OpenCore to support newer macOS versions on unsupported Macs. I wonder if it has a mode for supported Macs? Well, I suppose OpenCore can be setup manually if it does not. You would have to manually replace the original Lilu and Whatevergreen.

Just be sure to zip any resulting files before posting to reduce post size.

 

[ninethirty]

screenshot of Apple menu -> About This Mac
I think the 5500M should support DSC but I'm not 100% sure.


You could use an external USB or Thunderbolt NVMe, install macOS and whatever else you like. Then use Migration Assistant to get all the apps and docs from the internal install.


Need output from AGDCDiagnose on the Intel Mac with AMD 5500 GPU to see if the display supports DSC and what the max DisplayPort link rate is. For a tiled display, it can get the EDID for both tiles.

/System/Library/Extensions/AppleGraphicsControl.kext/Contents/MacOS/AGDCDiagnose -a > AGDCDiagnose_a.txt 2>&1

On the M1 Mac, can you run the script at https://gist.github.com/joevt/e862b0088ef58b9144877d01401bcee8
This will show if the M1 Mac is using a tiled mode or a single tile mode.

Also on the Intel Mac, try a USB-C cable instead of a Thunderbolt cable and redo the AGDCDiagnose command. A USB-C cable will limit the connection to 20Gbps which is sufficient for 5K60 only if DSC is supported. The point of the USB-C connection is to eliminate the ability to have two DisplayPort connections over Thunderbolt which will remove the ability to do tiled mode. But in the case of a 20 Gbps connection, it is still Thunderbolt. For a pure DisplayPort connection, a Belkin Charge and Sync cable is best.

Take a screenshot of the Display Information tab for the display in SwitchResX for each connection type to see if the frequency range changes in each case.

I haven't installed Monterey 12.3 yet so I don't know what new override files exist for the Studio Display.
/System/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610
There's probably a new .mtdd file to describe the tiled display (named something like DisplayProductID-ae3a.mtdd ). It's only needed if you need tiled mode to get 5K60. It may be sufficient to copy that from Monterey 12.3 to earlier macOS versions. Catalina would be easy since you can simply make the root volume writable temporarily. The process is more difficult for Big Sur and Monterey. I don't know if the .mtdd file can work from the /Library/.../DisplayVendorID-610 folder instead of /System/Library/.../DisplayVendorID-610 folder.

The AllRez command ( #177 ) can list all the modes (including pixel format info such as DSC, or chroma sub sampling - this info depends on the GPU drivers - older drivers don't provide pixel format info). It also shows DisplayPort and EDID info like AGDCDiagnose (but AGDCDiagnose may be better at getting EDID). And it includes frequency range info. It currently works best on Intel Macs but produces some output on an M1 Mac which I can use to produce better output for M1 Macs.

Catalina might be best for testing DSC support (if AGDCDiagnose says DSC is supported and it won't work in Big Sur or Monterey).

For Intel Macs, I have a fork of the Lilu and WhateverGreen kexts which can do the patch that is enabled by the -cdfon boot-arg for all versions of macOS. The patch stops CoreDisplay in WindowServer from filtering out some display modes. It might be interesting to see what modes those are (use AllRez to list before and after the patch). The usual way of using Lilu and WhateverGreen is to use OpenCore to inject them. Installing them to /Library/Extensions might work, or they might not load early enough (before WindowServer starts). In that case, the kill command can restart the WindowServer process which might then allow the patch to be made. If you use any other Lilu based kexts, then you may need to recompile them using my kext.
https://www.insanelymac.com/forum/topic/321371-lilu-—-kext-and-process-patcher/?do=findComment&comment=2778421
https://github.com/joevt/Lilu
https://github.com/joevt/WhateverGreen
OCLP is a method to automatically install OpenCore to support newer macOS versions on unsupported Macs. I wonder if it has a mode for supported Macs? Well, I suppose OpenCore can be setup manually if it does not. You would have to manually replace the original Lilu and Whatevergreen.

Just be sure to zip any resulting files before posting to reduce post size.

Sorry, I got a little lost here.

It's not an option for me to take my files form my work computer and work on the M1 Mac. What would you suggest/like me to do on the Intel Mac to get you any info you're after? I can try a USB-C cable, but I was unclear on what info you'd want after trying that.

Thank you!

 

[joevt]

Sorry, I got a little lost here.

Start with the AGDCDiagnose command.

 

[ninethirty]

Start with the AGDCDiagnose command.

Gotcha, sorry. Ok, here's two zips. One with the Thunderbolt cable diagnostic report and screenshot of SwitchResX, and then the same things for a USB-C Cable. Thanks for your help!

 

[ninethirty]

I haven't installed Monterey 12.3 yet so I don't know what new override files exist for the Studio Display.
/System/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610
There's probably a new .mtdd file to describe the tiled display (named something like DisplayProductID-ae3a.mtdd ). It's only needed if you need tiled mode to get 5K60. It may be sufficient to copy that from Monterey 12.3 to earlier macOS versions. Catalina would be easy since you can simply make the root volume writable temporarily. The process is more difficult for Big Sur and Monterey. I don't know if the .mtdd file can work from the /Library/.../DisplayVendorID-610 folder instead of /System/Library/.../DisplayVendorID-610 folder.

When I connect my M1 Mac, I'm able to find the file you mentioned above. Should I attempt to place it in the same folder, or the other one you mentioned on my work Mac? Could this have any catastrophic consequences?

 

[Amethyst1]

Should I attempt to place it in the same folder, or the other one you mentioned on my work Mac?

I'd try both folders: first one, then the other.

Could this have any catastrophic consequences?

The worst thing that can happen is that things go wonky once the Studio Display is connected to the MBP. In that case, disconnect it and remove the files.

 

[ninethirty]

screenshot of Apple menu -> About This Mac
I think the 5500M should support DSC but I'm not 100% sure.


You could use an external USB or Thunderbolt NVMe, install macOS and whatever else you like. Then use Migration Assistant to get all the apps and docs from the internal install.


Need output from AGDCDiagnose on the Intel Mac with AMD 5500 GPU to see if the display supports DSC and what the max DisplayPort link rate is. For a tiled display, it can get the EDID for both tiles.

/System/Library/Extensions/AppleGraphicsControl.kext/Contents/MacOS/AGDCDiagnose -a > AGDCDiagnose_a.txt 2>&1

On the M1 Mac, can you run the script at https://gist.github.com/joevt/e862b0088ef58b9144877d01401bcee8
This will show if the M1 Mac is using a tiled mode or a single tile mode.

Also on the Intel Mac, try a USB-C cable instead of a Thunderbolt cable and redo the AGDCDiagnose command. A USB-C cable will limit the connection to 20Gbps which is sufficient for 5K60 only if DSC is supported. The point of the USB-C connection is to eliminate the ability to have two DisplayPort connections over Thunderbolt which will remove the ability to do tiled mode. But in the case of a 20 Gbps connection, it is still Thunderbolt. For a pure DisplayPort connection, a Belkin Charge and Sync cable is best.

Take a screenshot of the Display Information tab for the display in SwitchResX for each connection type to see if the frequency range changes in each case.

I haven't installed Monterey 12.3 yet so I don't know what new override files exist for the Studio Display.
/System/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610
There's probably a new .mtdd file to describe the tiled display (named something like DisplayProductID-ae3a.mtdd ). It's only needed if you need tiled mode to get 5K60. It may be sufficient to copy that from Monterey 12.3 to earlier macOS versions. Catalina would be easy since you can simply make the root volume writable temporarily. The process is more difficult for Big Sur and Monterey. I don't know if the .mtdd file can work from the /Library/.../DisplayVendorID-610 folder instead of /System/Library/.../DisplayVendorID-610 folder.

The AllRez command ( #177 ) can list all the modes (including pixel format info such as DSC, or chroma sub sampling - this info depends on the GPU drivers - older drivers don't provide pixel format info). It also shows DisplayPort and EDID info like AGDCDiagnose (but AGDCDiagnose may be better at getting EDID). And it includes frequency range info. It currently works best on Intel Macs but produces some output on an M1 Mac which I can use to produce better output for M1 Macs.

Catalina might be best for testing DSC support (if AGDCDiagnose says DSC is supported and it won't work in Big Sur or Monterey).

For Intel Macs, I have a fork of the Lilu and WhateverGreen kexts which can do the patch that is enabled by the -cdfon boot-arg for all versions of macOS. The patch stops CoreDisplay in WindowServer from filtering out some display modes. It might be interesting to see what modes those are (use AllRez to list before and after the patch). The usual way of using Lilu and WhateverGreen is to use OpenCore to inject them. Installing them to /Library/Extensions might work, or they might not load early enough (before WindowServer starts). In that case, the kill command can restart the WindowServer process which might then allow the patch to be made. If you use any other Lilu based kexts, then you may need to recompile them using my kext.
https://www.insanelymac.com/forum/topic/321371-lilu-—-kext-and-process-patcher/?do=findComment&comment=2778421
https://github.com/joevt/Lilu
https://github.com/joevt/WhateverGreen
OCLP is a method to automatically install OpenCore to support newer macOS versions on unsupported Macs. I wonder if it has a mode for supported Macs? Well, I suppose OpenCore can be setup manually if it does not. You would have to manually replace the original Lilu and Whatevergreen.

Just be sure to zip any resulting files before posting to reduce post size.

In the case of the first folder /Library/.../DisplayVendorID-610, that folder didn't exist. There were no folders. I created that folder and added the file.

In the case of the second folder /System/Library/.../DisplayVendorID-610, that folder exists, but I cannot add the file. It won't let me. When I inspect the folder and try to change rights to read and write, it says I don't have permission. I have admin access on this machine, but it's not letting me do it.

Edit: On restart, no change with the monitor.

 

[Studiodisplay44]

I have the exact same issue. I have a 2020 13 inch intel MacBook Pro which is a work computer. I also have a m1 Mac mini. My m1 mac mini displays at native 5k as expected. My intel MacBook Pro can only output at 4k. I had my IT department update my macbook pro from big sir to Monterey 12.3 and the issue remains. I have also updated the studio display firmware.

I have been in contact with Apple who have taken a system diagnosis and will come back to me.

My guess is that this issue will be affecting all intel Mac’s used with the studio display. It hasn’t yet become so wide spread as the monitor is still so new with limited availability and secondly because the issue is quite subtle and many will not be aware (4k still looks sharp and everything else works as expected). In time I suspect this or another thread will continue to grow.

 

[Amethyst1]

I have the exact same issue. I have a 2020 13 inch intel MacBook Pro which is a work computer.

Can you provide AGDCDiagnose‘s output (see post #26)?
How did you determine that the Studio Display is driven at “4K” resolution?

 

[joevt]

Ok, here's two zips. One with the Thunderbolt cable diagnostic report and screenshot of SwitchResX, and then the same things for a USB-C Cable. Thanks for your help!

I looked at the AGDCDiagnose files:

• The 1st tile EDID has timings for 1080p, 1440p, 4K, 5K. It also has a timing for the 2560x2880 tile used for dual tile mode. All expected for a dual tile display. The new thing with the Apple Studio Display is that it includes a 5K 60Hz single tile mode (a custom timing can add a 5K 39Hz single tile mode for the LG UltraFine 5K).
• Only the first tile connection is used (port 4 HBR2 x4) in these examples. DSC is supported (Reg: 000060) by the display but is not being used (Reg: 000160)
• The 2nd tile EDID only has timing for the 2560x2880 tile. This means the display does not support a picture by picture mode like some other dual tile displays do (like the Dell UP3218K or LG 5K2K).
• The 2nd tile connection only exists with the Thunderbolt cable as expected but it is not a valid connection (port 5 status says 0 lanes link width, RBR link rate). This connection doesn't support DSC.
• The display does not appear to support HBR3 link rate (Reg: 002201) so it can't support 5K 60Hz 8bpc without DSC.
• AGDCDiagnose seems to indicate that DSC is supported on the 5500M. I don't know why it's not getting used.
• Only 3840x2160 60Hz 529.19MHz RGB 10bpc SDR is being output (framebuffer scaled down from 5120x2880)

There are three paths to getting 5K (I have tried none of these):

• Try a lower refresh rate (to support single tile mode from any GPU that supports 5K width):
  • Use SwitchResX to create a 5K timing that can fit in HBR2 x4 bandwidth limit, such as 5K 47Hz for 8bpc or 5K 35Hz for 10bpc (using CVT-RB v2). Windows might be able to do 5K 60Hz using 6bpc (at least that works with the XDR).
• Get DSC to work (to support single tile mode from a GPU that supports DSC).
  • Try a different macOS version. Catalina might have better DSC support.
  • Otherwise a patch needs to be found for the current macOS to enable DSC.
  • Maybe add the following to an override file:
    

            <key>DisplayPort</key>
                <dict>
                     <key>DisableDSC</key>
                     <integer>0</integer>
             <key>DisableFEC</key>
                     <integer>0</integer>
                </dict>

• Get the .mtdd to work (to supported tiled mode through Thunderbolt 3). If you're not using Monterey 12.3, then you need a new .mtdd file. If a .mtdd file doesn't work from /Library and needs to be in /System/Library, then you need a method to alter /System/Library or a patch.
  • I would try from Catalina, since you can easily modify /System/Library by temporarily mounting the root directory as writable. Maybe the file format of the mtdd has changed between Catalina and Monterey? In that case, you might need to model the .mtdd file after what exists in Catalina for the XDR display ( DisplayProductID-ae2e.mtdd ).
  • For a patch, I would make a Lilu/WhateverGreen patch to load all overrides from /Library/ instead of /System/Library.
  • For Big Sur and Monterey, there is a method to modify /System files. OCLP uses the method to apply post install patches. I would try this a single time to create a symbolic link between /System/Library/Displays/Contents/Resources/Overrides to a new folder named /Library/Displays/Contents/Resources/SystemOverrides or something like that.

 

[joevt]

• Try a lower refresh rate (to support single tile mode from any GPU that supports 5K width):
  • Use SwitchResX to create a 5K timing that can fit in HBR2 x4 bandwidth limit, such as 5K 47Hz for 8bpc or 5K 35Hz for 10bpc (using CVT-RB v2). Windows might be able to do 5K 60Hz using 6bpc (at least that works with the XDR).

Did you try this yet?

• Get the .mtdd to work (to supported tiled mode through Thunderbolt 3). If you're not using Monterey 12.3, then you need a new .mtdd file. If a .mtdd file doesn't work from /Library and needs to be in /System/Library, then you need a method to alter /System/Library or a patch.
  • I would try from Catalina, since you can easily modify /System/Library by temporarily mounting the root directory as writable. Maybe the file format of the mtdd has changed between Catalina and Monterey? In that case, you might need to model the .mtdd file after what exists in Catalina for the XDR display ( DisplayProductID-ae2e.mtdd ).
  • For a patch, I would make a Lilu/WhateverGreen patch to load all overrides from /Library/ instead of /System/Library.
  • For Big Sur and Monterey, there is a method to modify /System files. OCLP uses the method to apply post install patches. I would try this a single time to create a symbolic link between /System/Library/Displays/Contents/Resources/Overrides to a new folder named /Library/Displays/Contents/Resources/SystemOverrides or something like that.

It appears that displaypolicyd is responsible for reading the .mtdd file (at least on Intel Macs). I believe it can load an .mtdd file from the /var/db/displaypolicyd/ folder. The .mtdd file for the Apple Studio display can be copied their from Monterey like this (change the names of the volumes to whatever you're using):

sudo cp /Volumes/Monterey/System/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610/DisplayProductID-ae3a.mtdd /Volumes/BigSur/var/db/displaypolicyd/Display-APP-ae3a.mtdd
sudo cp /Volumes/Monterey/System/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610/DisplayProductID-ae3e.mtdd /Volumes/BigSur/var/db/displaypolicyd/Display-APP-ae3e.mtdd

There's a log file in the /var/log/displaypolicy/ directory corresponding to your GPU (using PCIe bus/device/function numbers in decimal) which should show if it reads the .mtdd file successfully (view it before and after you connect the display).

I don't know if you also need the override files:

sudo mkdir -p /Volumes/BigSur/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610
sudo cp /Volumes/Monterey/System/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610/DisplayProductID-ae3[a-f] /Volumes/BigSur/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610

If you try these, remove the override file created by SwitchResX so SwitchResX can recreate a new one.

There's also a DisplayProductID-ae3f.tiff file containing an icon for the Apple Studio Display but that requires editing the Icons.plist file in the Overrides directory...

 

[joevt]

Here's a post showing DSC working from a 6600 XT to the Apple Studio Display via a Thunderbolt add-in card.
https://egpu.io/forums/gpu-monitor-...5k-thunderbolt-monitor-with-egpu/#post-100745

 

[Amethyst1]

Here's a post showing DSC working from a 6600 XT to the Apple Studio Display via a Thunderbolt add-in card. […]

Do we know when its ae3b through ae3f product IDs show up?

 

[joevt]

Do we know when its ae3b through ae3f product IDs show up?

ae3a is for the tile1 or single tile EDID and the tile2 EDID.
ae3b is the overlay EDID from the ae3a .mtdd file.

I don't know when ae3e is used. The override and .mtdd info are the same for ae3a except the product IDs.
ae3f is the overlay EDID from the ae3e .mtdd file.

All four have the same Icons.plist info.

 

[varen]

Just to chime in on this, I wanted to report I'm having no problem running my Studio Display at its 5K resolution on my 2019 MBP 16" with a 5500M discrete GPU. When plugged directly into my MBP.

I AM however limited to 4k resolution if I try connecting the Studio Display to my Blackmagic eGPU Pro's 2nd thunderbolt port. This thread has given me some amazing info on how to dig into and diagnose that. Thank you!

 

[joevt]

Just to chime in on this, I wanted to report I'm having no problem running my Studio Display at its 5K resolution on my 2019 MBP 16" with a 5500M discrete GPU. When plugged directly into my MBP.

Thunderbolt 3 cable connected at 40 Gbps x1? Is it using HBR2+DSC or dual HBR2? Do you get USB 2.0 speed or USB 3.x speed?

I AM however limited to 4k resolution if I try connecting the Studio Display to my Blackmagic eGPU Pro's 2nd thunderbolt port. This thread has given me some amazing info on how to dig into and diagnose that. Thank you!

Blackmagic should be able to use dual HBR2 mode like it does for LG UltraFine 5K. You are running latest macOS Monterey? It is connected at 40 Gbps x1?

 

[varen]

You are running latest macOS Monterey?

No... I'm still running Catalina which I realize now is probably the problem! System Profiler shows that the eGPU Pro and the Studio display are both at a speed of "Up to 40 Gb/s x1" and both the cable in the computer -> eGPU -> Studio Display should be proper Thunderbolt cables. When I connect a USB-C drive to the Studio Display's USB-C ports I see it's connecting at "Up to 10 Gb/s" in System Profiler. (I tried using a USB-C cable for eGPU->Display to force it into a different mode, and the display still worked, but was also still stuck at 4k)


I'm 98% sure my problem will just be fixed if I upgrade to Monterey! But if curious here's a bunch of debug info: AGDCDiagnose, switchresx screenshot and EDID export, and the resolutions shown by my favorite `displayplacer` tool. Cheers!

 

[joevt]

No... I'm still running Catalina which I realize now is probably the problem! System Profiler shows that the eGPU Pro and the Studio display are both at a speed of "Up to 40 Gb/s x1" and both the cable in the computer -> eGPU -> Studio Display should be proper Thunderbolt cables. When I connect a USB-C drive to the Studio Display's USB-C ports I see it's connecting at "Up to 10 Gb/s" in System Profiler. (I tried using a USB-C cable for eGPU->Display to force it into a different mode, and the display still worked, but was also still stuck at 4k)


I'm 98% sure my problem will just be fixed if I upgrade to Monterey! But if curious here's a bunch of debug info: AGDCDiagnose, switchresx screenshot and EDID export, and the resolutions shown by my favorite `displayplacer` tool. Cheers!

I see that both tiles of the Apple Studio Display are connected to the Blackmagic eGPU so there's no problem with the cable except the second tile is set to RBR link rate with 0 lanes which maybe just means its not being used for the 3840x2160 529.19 MHz RGB SDR 10bpc mode that is currently being used for the scaled 5K framebuffer resolution. And of course the Blackmagic eGPU doesn't support DSC. Catalina won't use the two tiles without a .mtdd file - maybe the one from Monterey 12.3 will work. I've added the Monterey files to #36 if you would like to try.

Before trying the .mtdd, try a lower refresh rate? Start with 5120x2880 30Hz CVT-RB2 and work your way up to 47Hz. Which refresh rates work? I think LG UltraFine 5K can do up to 39Hz. After trying this, delete the override file created by SwitchResX and try the .mtdd.

I would also like to see an AGDCDiagnose with the Studio Display connected to the 5500M since it supports DSC.

 

[Scott Robinson]

Hm. That's a 3840×2160 mode with a "Looks like 2560×1440" HiDPI mode on top of it, not a 5120×2880 one.

Try deleting your custom 5120×2880 mode and recreating it with the following settings taken straight from the display's EDID:

View attachment 1976671

The greyed-out Blanking and Total rows will be filled by SwitchResX; fill in the other values as shown in the screenshot but use a point . as the decimal separator for Pixel Clock and Scan rate (my macOS is normally set to German which is why there's a comma instead, sorry!).

File > Save settings and Activate immediately. Test the mode. If it works (everything will be tiny but what matters is: does the Studio Display display it properly?), set the Scaled resolution base to 5120×2880 and reboot. Try the 2560×1440 HiDPI mode next, double-clicking to check if the Active row reads 5120 pixels by 2880 lines.

MacBook 2018 15” here with the same issue. The manual settings described here worked for me. Thank you.