Latency using Synth instruments on GarageBand?

I did a search but couldnt find anything with my specific problem.

Im using GarageBand 09, with an M-Audio Fast Track Pro. When I plug in my guitar directly and use one of the amp models there is no latency (noticeable at least)
I've recorded a guitar track with the m audio unit and my vocals with a Blue Bird condenser mic. Tried recording the guitar by itself and guitar and vocals simultaneously and it works fine. but....

The problem is when I try to use my MBP keyboard to play "drums" I get a good deal of latency. Why is that? Is the onboard sound card not good enough for that? Should I get a dedicated MIDI/keystation to play those Synth instruments and run it thru the M-Audio Fast Track Pro.