Me too
Yes, I have the same problem. I tried copying the "Terminal.app" program across from a 10.5.8 Mac, and although this functioned OK (despite being the wrong OS version), it also cached the password.
So this would appear to be the OS not the Terminal program. However, I can't find anything in the KeyChain which would explain this behaviour. I'm not sure if caching is the right word, because it still does it even after a reboot.
Re other messages in this thread, I too am logging into a server which NEEDS an SSH rsa key authentication (SSH password not accepted). I find this odd because I believe Apple uses fairly standard OpenSSL/SSH software, so they must have set the default config options differently.