[Edit: Ninja'd - sounds like it was stuff in your trash, which would have shown up in the du command below as ".Trash". One possible cause for things showing as bigger than they should in the Finder after deleting stuff is that it hasn't updated its cached file info yet--logging out and back in might fix that. Try it before you go chasing phantom files.]
If I'm in the folder itself and get info it says 79GB
You're saying that if you do a select-all when inside your user folder and hit command-option-i, to get the multiple-item Info window, it says 79GB?
In which case, whatever is taking up the phantom space is a file that's not visible in the Finder; probably a .something directory that isn't shown by the Finder.
The easiest way to do a check is one of the apps that GGJstudios recommended.
If you want to try the terminal, you can open a terminal window (which should default to your user directory), and type "sudo du -sh *", type in your password, and see if that matches what you're seeing in the Finder. If so, try "sudo du -sh \.*" which should show all the .something directories; see if any of those are huge, and if so, which one.
