Sure, Cydia itself is slow, but it slows down the device overall? I don't see how -- it's not like it runs in the background or otherwise takes up system resources while it's not running. Personally, I've never noticed any slowdowns on my jailbroken iDevices. A bit faster battery loss, yes, but system slowdown, no.
It's everything else u downloaded from cydia like appsync, always needed to run cracked apps. Themes etc