usually you want to use the programs uninstaller, if it has one. Otherwise you may have to take a trip to the "library" folder and investigate what the program left behind in terms of startup items, preference panes, caches etc.
Thats true of some apps. But Adobe CS3-CS5 for instance sprinkle stuff in almost every conceivable place in the library folder.
But yeah, for smaller apps, not including audio or video content authoring apps, that is generally true.
OTOH some companies like MAudio, dont put anything in the app folder at all when you install the driver for their USB keyboards. There is just a little guy they put in the midi folder in the library. So you are left scratching your head, did it go?