It really means I am consistently in control of whether to leave a program open in the background or not. I know ⌘q will *always* clear that program from working memory[1] (like Windows default, except Windows does not IIRC have a consistent key for quitting apps), and ⌘w will *always* close the document window but leave the program fully functional in memory. Traditionally in OS X the (x) icon is linked to ⌘w, which leaves programs in memory and let the OS manage memory; using ⌘q allows me to force a program out of working memory, a kind of "hint" to the virtual memory system of my working habits.
In regards to performance issues, most Apple users use the (x) icon and don't have problems, the OS virtual memory system is based on very general methods developed across unix and windows systems for keeping the system running optimally. Some people prefer to keep most of their memory free, which I personally find ridiculous, and I think a computer scientist would generally agree.
----
[1] actually memory is marked as inactive, so restarting that program is still faster than starting from cold.