Mac or PC? Either way if you've just installed VLC, you need to open it once first to get past that "authorize opening a downloaded application" screen thing. Then Handbrake will be able to access the codecs and stuff from VLC.
I have a PC. Thats weird bc I had VLC open will I was using handbrake for elf. I was able to rip the first 5 min of elf using vlc, but dont know how to rip the whole movie.