Playing a 1080P video on a rMBP

I am no entirely sure how VLC handles this, but if it has proper retina support, it should render the full 1080p video which gets upscaled to 2880x1800

It will play the video in 1080p since there are more then enough pixels, blacks bars will be present since retina isn't 16:9 but 16:10. On full screen, it will just try to fit 1080*1920 pixels on the retina screen (up scaling)On a non retina it will downscale the video since there aren't enough pixels to show all 1080*1920 pixels. Again with black bars since 1080p is 16:9 and Macs have 16:10 screen ratio.

Even if you're at the best for Retina setting, it will always display photos/videos, etc. at an independent resolution. So yeah, 1080p.

Might be beneficial to use some post processing if you are not on battery and the system can handle the load.
VLC has some software alternative scaling methods and filters to do part of what any decent picture processor in your TV does.
My system can only handle it on 720p material but with post processing you can get 720p material to look close to 1080p stuff on a 1080p native monitor.