http://mroogle.*************/MRoogle.gif can answer many questions.
First, Handbrake does not really rip* the DVD, it just transcodes the video to a new format and new codec. It uses the VLC Player to circumvent the copy protection scheme, so if Handbrake is 64-bit, VLC Player needs to be 64-bit.
Second, do you really press the RIP button, or does it just say START? Is there maybe some other software involved?
You can also help us figuring out your problem by telling us what exact Mac you have (Applications / Utilities / System Profiler > Hardware Overview) and what Mac OS X version you run on it

apple. > About This Mac).
Also what Handbrake and VLC version do you have?
And what settings in Handbrake did you use?
What does Handbrake say when it performs the "first step"? Is it a two pass calculation?
Have you taken a look at Activity Monitor (select to SHOW ALL PROCESSES and sort by CPU) and seen how much CPU Handbrake uses?
* Ripping a video DVD means that the entire video DVD and its contents are copied onto the HDD while the copy protection scheme is circumvented.
The copied video DVD is an exact image of the video DVD, and contains an AUDIO_TS and a VIDEO_TS folder.
Transcoding is the conversion of one format with one codec to another format with another codec.