Even though my rMBP has 16GB RAM, it frequently taps in to swap memory. So I started cracking down on apps and processes running the background. One oddity was "com.apple.speech.recognition" eating up nearly 1GB of physical memory (even after a reboot). So I turned it off completely by going to "System Preferences | Dictation & Speech | Dictation -> Off & Shortcut -> Off". Voila, I got 1GB RAM back.