I had the exact same problem with my 2010 MacBook Pro connecting to a 24 inch LG LCD monitor. Using a DP - HDMI cable worked, but it didnt produce sound. Using an adaptor worked for sound, but gave me the weird resolution problems.
Here's my thread on this (its pretty detailed);
https://forums.macrumors.com/threads/1203042/
The solution? Well, at first I thought it was a fault with the laptop, I brought it in and demonstrated the problem to the apple tech guys in store. They replaced the mobo on it. I was still having the same problem. They replaced it again, only this time the laptop was freezing randomly (in OS X, in Windows and even when trying to install Windows 7), but on top of that, messed up my serial number when they inputted it (it was fine on the laptop, but not in OS X). So they just gave me a brand new 2011 MacBook Pro. But, I was still having the same problem.
I sold the old monitor and got a Dell Ultrasharp U2410. Using a DP-HDMI cable works, but no sound. So, I have to use an adaptor from DP-HDMI and then run a male to male HDMI cable to the monitor and it works fine now.
The weird thing is this though;
When I was running the old 2010 MacBook Pro in Windows 7, using the stock drivers from Apple's OS X DVD gave me the same problems as in OS X. But, when I updated the Geforce 330M drivers manually, all cables worked perfectly fine in Windows.
So, its definitely a problem with Apple's drivers. But I wasn't surprised, their choice of video cards for Mac Pros is an insult to professionals, and their OS X drivers with the lack of OpenGL 4.0 this is even more of a joke.