Personally I would never carry my DSLR while I go mountain biking; my riding style isn't that of going downhill to the extremes, more cross country endurance riding, but I'd always feel weary knowing that an expensive piece of gear is at the back; I have a solid camera backpack but if I fell it'd still hit a lot of force.
If you have an iPhone 4, just do what i do and buy an armband and you'll have your camera at easy reach. Previous gen iPhones same applies but given the lower quality of pictures might want a point and shoot instead.
I dont know about other peoples opinions, but when Im out riding, taking pictures would remain a second priority of mine; enjoying the ride would come first, as such I wouldn't want to have to ride slower than usual, or more cautious than usual - I don't want to always be thinking 'dont go to fast or something - important gear at the back!'.
Your concern about not enjoying the ride is sensible. It depends upon the rider if one is hardcore or not. I will try both using a SLR and a P&S. I am not a street rider, as I ride on trails etc.. The P&S is clearly the winner with the iPhone 4 being a close second place. I have one of the two covered.