To work with an ATV3 then you should have an HDTV capable of supporting 720p or 1080 as listed on the Tech Specs page.
Nowhere in the tech specs of the ATV3 does it state that it supports using Monitors at other resolutions.
Apple make the product for working with HDTV's and when plugged into a HDTV either 720p or 1080p then they work fine.
1680 x 1050 is a 16:10 ratio rather then a 16:9 as found on Widescreen TV's, so I am not surprised that is not recognised by the ATV3.
Apple gear is not perfect in that you can't connect it to whatever you want to and have it work at the best possible resolution, and the AppleTV3 is probably not the correct solution for your requirement.
I find Apple equipment works fine when being used as Apple intended. It tends to not work as well when people use it outside the environment that Apple intend the product to work within.
Being hooked up to non-TV resolution monitors not being something that Apple intend the ATV3 to be done.
Apple is no different to other manufacturers in this kind of thing.
I own a ATV mk1 which I connected to a Samsung R74 TV with the HDMI cable. It won't display iTunes media that is content protected as the ATV does not find any HDCP, so instead I am forced to run via component. I could select 1080i but on a 1366 x 768 then the 720p looks better to my eyes.
When I raised this with Samsumg then the Support answer was that Apple had implemented a non-standard HDMI interface and that the problem was Apple's. This despite it working fine on an HDMI monitor. It turns out that Samsung simply didn't implement HDCP on the HDMI ports on the R74 tv sets, yet it is apparently Apple's fault that they don't work together, and the support team would not acknowledge that no HDCP on the HDMI interface, I had to dig this out on forums on the Internet myself.