I'm completely new to Time Machine - haven't used it at all yet. And I'm just wondering, does it work with a regular USB external hard drive? Or can I only use it if I buy a wireless Time Capsule?
yes, Time Machine works with a normal USB hard-drive.
to configure it open System Preferences, go to time machine, click on choose backup disk and just choose the disk that you want!