The previous posters are right. The two best options for you are:
1. Use MacTheRipper and rip the entire DVD (usually ~4 or ~8 GB) onto your HDD. Next open Apple's DVD player, then File > Open DVD Media > the VIDEO_TS folder inside your ripped DVD folder. Another easier way is to simply drag the desired VIDEO_TS folder onto the DVD Player icon in the dock. There is also software out there (
http://www.macupdate.com/info.php/id/20141, I have never used this software, use at your own risk) that adds extensions onto the ripped folders and then automatically opens DVD Player when these files are opened.
2. Use HandBrake and rip the video into .avi or .mp4. This will save LOTS of space and retains quality video and sound, but you will lose all menus, subtitles (unless ripp them into the video file), and extras. It also takes a lot more CPU/battery power to play highly compressed .avi or .mp4 files. And worst of all, it takes lots of time to rip the videos, around 3 hours for most movies on my iBook G4.
Hope this helps. If there really is some incompatibility issue between MTR and your DVD, use the HandBrake option