Before you do anything, backup the entire directory and also backup the database. There are plugins for this, but I find it easiest to just use FTP for the entire contents of the directory (make sure hidden files are included), and phpmyadmin for the sql.
Move the files to the new directory or server. Upload the sql if you need to, and edit the wp-config file to reflect new location/settings.
Please be aware that some images or older plugins may have absolute paths that won't work after the move. The easiest way to solve this is to install the "Search and Replace" plugin for wordpress and look for your old absolute path and update to the new location.
I've done it dozens of times. It takes minutes... just make sure you backup first in case something goes wrong.
