This is WEP, yes? And you're using Leopard, right?
Have you tried the "WEP Password" option and just used the Hex key?
(as of now) I don't know if that will work or not.
EDIT: Ah, now I know... Choose the "WEP Password" and put a dollar sign ($) before the hexadecimal password.
FYI, 40 bit = 64 bit. It's an industry nomenclature thing.