Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
I'd go with a refurbished CD mini, for $430 (for the lower end one) it's a very able machine. The downside is the lack of a superdrive, which is pretty huge.

You can probably find a free or really cheap CRT (<$20), and a mouse and a keyboard can be gotten for under $50. That's about $500.

A good suggestion for under $30 on ebay for mouse and keyboard is the mac specific logitech S530.

Then later on you can buy a LCD, and maybe a superdrive or external dvd burner drive.. but you are probably better just saving for like a refurbished iMac.
 
mac mini or an old iMac are your best option (depending if you need a screen or not)
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.