this "issue" is supposed to be a feature

when in fact it´s annoying as hell...
the absolute first thing I changed on my rMBP was to hibernationmode 0
check which mode is on (terminal):
pmset -g | grep hibernatemode
and you set mode 0:
sudo pmset -a hibernatemode 0
0 - Old style sleep mode, with RAM powered on while sleeping, safe sleep disabled, and super-fast wake.
1 - Hibernation mode, with RAM contents written to disk, system totally shut down while “sleeping,” and slower wake up, due to reading the contents of RAM off the hard drive.
3 - The default mode on machines introduced since about fall 2005. RAM is powered on while sleeping, but RAM contents are also written to disk before sleeping. In the event of total power loss, the system enters hibernation mode automatically
----------
in hibernationmode 0 your machine shuts down at a total powerloss (so you know u can lose data, if you ignore all warnings)
EDIT: oh and the slow wake up has nothing to do with powernap, your wifi, FileVault or anything... its all bout the hibernation