I've gotten some of my best pictures and videos with my phone camera, simply because I have it on me wherever I go. I don't think anyone will argue that a cellphone or even a high end point and shoot like the S95 can come close to competing with the lowliest of SLR cameras. But given the size, they are a pain in the ass to carry around. I even find my Canon P&S, which fits in my front pocket, a pain to carry around most of the time.
If you don't mind hauling around the big camera, then go for it, as it'll give you the better quality. However, with proper composition and lighting, you can capture some amazing moments with the small camera that you have on you at all times.