Taking a look at the logs in /var/log could shed some light on the problem. Be forewarned, they're not the most fun thing to read. Look for stuff referencing sshd.
Here's what I found:
Mar 25 10:28:32 Mac-Pro sshd[41215]: USER_PROCESS: 41215 ttys000
Mar 25 10:28:32 Mac-Pro com.apple.launchd[1] (0x10d960.sshd[41215]): Could not setup Mach task special port 9: (os/kern) no access
Mar 25 10:34:44 Mac-Pro com.apple.launchd[1] (0x10e160.sshd[41252]): Could not setup Mach task special port 9: (os/kern) no access
Edit: Here's what happens when I start SSH in System Preferences:
Mac-Pro com.apple.launchd[1] (com.openssh.sshd): Unknown key: SHAuthorizationRight