My apologies, it appears that multiple instances feature is only in the Windows version of VLC. (I used VLC back when I lived in the dark side)
It appears to be submitted on the feature request list for the OSX version - but I'd browse the videolan forums / wiki for more information on it.
Didn't intend to mislead you there.