On Retina it's much better to set your desktop resolution to 1440x900 or some other you want to play and then start dota and set that resolution. But standard resolution no HiDPI. And when you quit Dota just set back to best for retina.
But you should consider installing bootcamp. I used bootcamp then moved to OS X but for now i'm back in bootcamp and must say OS X is not a system for good Dota gaming. And it's not even about performance. More about wicked mouse sensitivity, hard to set properly, slowdowns here and there.
Just install bootcamp, then bootcamp in OS X (an icon so you click and it restarts into bootcamp)
It's much faster to restart into Bootcamp and start Dota in Windows then setting mission hot corners off, setting resolution and starting dota. Trust me😉
In bootcamp with retina you get 1080p max detail at 45-60 fps. If you turn off world lighting and specular you get around 90. And Ti4 is coming so you can watch games in 2880x1800 at 30 fps😉 (it's good for watching but not best for playing)
cheerz