Correct - each application on a Mac is just a folder with a .app extension. Like you noted, the only things that aren't in that folder are caches, preference files, and other supporting files (all typically pretty small). They're fairly self-contained and can typically be removed just by removing the .app folder.
Some people (including me) really like to remove the plist, cache and app support files as well. I use AppZapper to do it, but you could just as easily use Spotlight and delete them yourself.