You would be right, it does take longer for SMS messages to travel across two carrier networks than it would to transport a message in-network.
The simple reason for this is, well, it's a different carrier. Unlike voice phone calls, where everything is pretty well standardized at the cross-network level, there's some translation and specialized routing that has to take place for an SMS message to cross a network boundary, at least in the US.
Further complicating matters is that while AT&T and T-Mobile mostly adhere to the same SMS stadards built in to GSM (and thus used by most everyone else in the world), the other two large carriers - Verizon and Sprint - use different networks running on different standards, and their SMS systems do not conform. Verizon is close, but Sprint is completely oddball in the system they've put together for their CDMA network, and doesn't even conform entirely to CDMA-established standards. And Nextel? Forget about it, it's a mess.
Because there are these differences, US carriers have to have technical and financial arrangements in place to allow intercarrier messaging. Some networks individually work it out with the other carriers, while others completely farm out the whole mess to a third party, like
VeriSign, and leave any out-of-network message in that third parties' hands to deal with. Along the way, the message has to be reformatted to fit the needs of the destination network, and checks have to be made to be sure that the number hasn't been ported out to a totally different network. This verification isn't always up to date either, which is why just-ported numbers can have problems receiving text messages for as along as a day or two.
The problem with this arrangement is pretty obvious: when messages don't get to where they need to go, it's hard to pinpoint who's to blame. It could be breakdown on AT&T's end, or it could be on the receiving end. It could that a lookup table for ported numbers wasn't updated on time. Or it could be the third party gateway having trouble. And obviously, no one wants to take the blame and will point fingers at the others.
Once you get into the deep details of what goes on, you'd actually be very surprised that any messages make it at all. But somehow, it all works... most of the time.