You bring up a good point. With these drives being smaller and smaller, encryption is key. I use a Seagate Blackarmor 320GB for work. It is hardware encrypted, and it is slightly larger than the passport, but protecting that data is key. If you are going to be carrying around personal data (like your Quicken checking file) either on the laptop or the external hard drive, I'd suggest encryption.
If all you plan to do is keeping media (movies, songs, etc.) not so much. Pictures, meh, not sure I'd care but some people might not like a thief to be able to see their pictures. This should be a consideration when making portable storage decisions.