where do I start

A Mac Mini or MacBook or 21.5" iMac will just do fine. If you order via Apple, make sure you look at the Refurbished Section.
If you order via some other place, look here for price comparison. Also take a look at the MR Buyer's Guide to get a feeling of imminent updates, but every Mac except the MacBook Air has seen an update since at least April.

Also have a look at the following links (if you haven't already), as the information presented there might be helpful in your future endeavours into Mac OS X and could clear up initial confusion and may even prevent harm to your system or your files.


Mac OS X Basics

Switch 101 - guide with articles made by Apple on how to accustom yourself, after you switched to Mac OS X from Windows​

Mac 101 - How to get started with Mac OS X​

Find out how - tutorial videos made by Apple on how to do certain thing in Mac OS X​

Pro tips - tips made available by Apple for easier ways of doing certain tasks​

Mac OS X Keyboard Shortcuts - Learn about common Mac OS X keyboard shortcuts.​

Mac OS X Beginner's Guide by MacRumors - learn about software, media players, shortcuts and some useful tips, tricks and hints​

Mac Guides - tutorials, product guides and more​

Mac Guides - Finding Mac Software​

MRoogle - a very effective tool to search these fora using Google and made available by edesignuk, introductory threads: 1, 2 and 3​

"How to maximise your MacRumors troubleshooting experience" created by mad jew in 2006​

 

^^ What he said. I recommend if you want portability, a regular refurbished MacBook (the white one) if they have them on the refurb part of the apple store online. If not, get a refurb baseline MacBook Pro.

If you don't need portability, get a refurb Mac Mini and use an external monitor, as this would be the cheapest route. If you prefer all in ones, a refurbished iMac would do more than suffice for your needs.