Is it possible For an app off the app store (not cydia) to slow down your iPhone 4?
If you explain a little more I can give you a precise answer.
Apps usually can't harm when they are not running. Even when in background (since they are actually not really running) except if it's getting GPS events or playing audio.
There are other things an app can do that will slow down the phone but they should be easy to see; for example an app can add 100,000 contacts to your address book; when you use the address book it will be slower but you should see the contacts...
Lastly there are things that may happen that YOU may think are caused by an app (maybe you have just finished using it) but really depends on other factors.