Have you tried looking inside the VIDEO_TS folder? Sorry if this sounds like a dumb question, but most often when I do the task you are trying to accomplish, the same kind of folder is created and the actual video is inside this folder. If the folder is empty, then I suppose there is some incompatibility issue between your DVD and handbrake.
I like to use Mac the Ripper to convert DVD's to digital files. You can download it for free from many places on the internet. I got mine from this website:
http://www.macupdate.com/info.php/id/14414/mactheripper
It has always worked wonderfully and I've never encountered any problems with it, nor have I ever gotten any viruses/spyware/etc from downloading it from that particular site.
The developer's website is no longer functional, so finding information about the program can be a bit tricky, but if you just search in forums such as this one or look for tutorials on youtube, you can usually find whatever info you need regarding this program.