Yes. OS X is designed so that you can launch applications and then never close them (just close the window - ⌘W or the red button). Then you will continue to get the notifications. Properly written applications consume little resources when not being used, so you can have dozens running with little impact.
----------
You have something wrong. I've got 5 POP, 2 Exchange, and an iCloud (IMAP) account and it barely blips the gauges except when checking mail. You can control the rate at which it does mail checks. If it is consuming all that CPU all the time you should try rebuilding the mailboxes.
I understand that, but other apps on my mac like Messages, App Store and other Apple apps are not supposed to be open in order to get their push notifications. For example, if someone sends me a message (SMS/iMessage) the Messages app pushes the message and I receive the notification even if the app is closed.
I guess that all these apps don't really close and something still runs in the background, but still, isn't there an email client that does the same..?