When you restore from TM, some OS cache files are not preserved. These might include copies of frequently accessed files in "hot areas" of the drive, which are the fastest parts of the drive. They are also things like the spotlight index, and the Mail database (I think... I don't use Mail). These files should be regenerated with time, and your boot time might go back down. I'm guessing it won't go all the way back to where it was, though, since a lot of files will be on a different place on the disk, and the OS doesn't optimize the location of every file.
If you still have the old drive any you haven't formatted it yet, you could try a low-level copy which will put everything in exactly the same place on the drive and keep any cache files.
