Generally speaking, you get what you pay for. Cheap cases usually mean that you have more than one manufacturer that copied the original design, so there's a chance you'll end up with a poor knock-off, as seen in the Vapor Pros from China thread.
I just bought a white bumper Apple clone from Amazon for $2.30 shipped. You can find it by searching for "iphone 4S bumper white". The holes may not line up exactly like the official bumper from Apple, but for $2 I'm ok with that. They also have a fun gameboy silicon case for $4.99.