Yes, if an app comes with an uninstaller, and Office 2004 does, you should use it to make sure you get rid of everything. Simply dragging and app to the Trash works for most small apps, but some apps like Office like to put their hooks into other areas of your system and you should get rid of those to be safe.
It should be in /Applications/Microsoft Office 2004/Additional Tools/Remove Office.