First of all many kudos to the OP for this thread. I can't figure out how it's not gotten enough replies, since (with a few caveats) it effectively helps solve on of the longest standing problems in OS X, that of non resolution independence. I 'll put the lack of attention down to the general low level of the apple forums these days.
To the guys having problems, I 've already set this up in a few macs, so guys it should work, just do it again and see if you might be doing something wrong, alternative try posting what you do step by step so we can help you out by spotting where you go wrong.
Now for the caveats, as far as I understand and correct me if I am wrong:
a. This might on principle work on every mac, BUT it will considerably degrade performance for any but the latest macs with the best gfx cards and cpu. What we are doing here is taking apple's (rather crude and nasty) hack for resolution independence by tricking the system to draw a twice the size canvas for the retina resolution we are aiming at and going into retina resolution then via system prefs. So as per the lagging and stuttering rmbp models the computation is costly, drawing at extra high dpi, then scaling the whole thing down to the existing screen size. The gfx needs to be able to support such high dpi rendering, and the cpu to be able to handle the computations for the scaling. On almost all macs you are definitely going to see slow frame rates, even on the retina pros the fps don't exceed 30fps, this is not for gamers that's for sure. Even the best gfx cards out there can't run the latest 4k monitors (sharp's for example) flawlessly at 60hz, so going for say a custom 3840 x 2160 via switchresx on the typical mobile gpus apple uses, or even worse the integrated intel *****, will take it's toll.
b. If I understand this right, the machine needs to have a high enough ppi for it to work, at least what the 27" inch has, if I remember correctly about 109ppi, because if it doesn't you are going to get less than perfect looking font. They are already less than perfect on apple's own implementation in the scaled modes on the retina models, so you can figure out how they are going to be looking in non retina macs. Of course they are not going to be near as bad as just running in a different resolution via system prefs or using the dreadful zoom in "feature".
Nevertheless this is a great hack via switchresx and again much, much kudos to the op. As far as I know this is the only place on the web showing you how to do that. And one question to the op: I am looking to buy the 27" imac, would you think I should opt for the highest end gfx for it to work you reckon, the NVIDIA GeForce GTX 680MX. What's your gfx and do you notice the system lag at all?
Sorry, I think you got some concepts wrong and maybe the OP too.
Let me try to explain it a little...
A resolution of 3800 (whatever) is actually not possible. When you set this resolution, what you actually do is to emulate the "look" of this resolution using the maximum number of pixels you might have in your monitor. Things will look tiny, that's all.
The problem is that only a bunch of some resolutions (apparently, only the Scaled ones) are allowed to be used by OS X when it boots (after you created them on SwitchResX), because Apple "pre-approved" them. There's some information about this on SwitchResX's manual, although it is not very clear about this, but they do state that many resolutions won't work after you reboot the system, mainly scaled ones.
The resolutions (scaled and HiDPI) available for all original Mac monitors and also 1080p ones are available with the look of half of the original resolution as "retina mode" or HiDPI. So, you can have a "retina resolution" for 1080p as the equivalent of 1920x1080p in 960x540p, for instance. Also for all of Apple's monitors and computers, with the Retina Macbook Pro with more options, as it is obviously the newest retina mac product.
The caveat is that you can't actually set other HiDPI modes that Apple hasn't enabled. You apparently can set different resolutions for your monitor if the hardware supports it, but you cannot set different HiDPI (or "retina") modes that are not originally available from Apple. That's why the OP's scaled resolutions which he "created", actually worked... Because they were already available by Apple.
Correct me if I'm wrong, but that's what I learned from two days with lots of research about the subject and practice too.
Try to set different HiDPI resolutions from the ones available from Apple... It didn't work for me. If it works for you, please send us a screen capture of both the settings and the resulting image. I tried to create a 1600x900 HiDPI mode to correspond with a slightly bigger UI than the original 1920x1080, but OSX does not accept this mode... It only has the 1440x900 HiDPI for 1920x1080 mode, but the sides are cut off, showing black bars... That's why I wanted the same HiDPI mode, but using the additional 160 "virtual pixels" to fill up the 1080p resolution.
Complicated? Well.. It is
