I used to use VNC as well, but leave the server off. I'd log in via ssh, start the VNC server, then connect. And only leave the server running as long as I was logged in.
I believe OSX has had a built in VNC server since 10.4, though I haven't used it.
It doesn't seem to me to be anywhere near the performance of (for example) Remote Desktop on Windows. I've never used the Apple Remote Desktop, so I don't know how it performs, but it doesn't seem to be aimed much at this type of work.