But your image circle is much smaller for the crop lens, and therefore designing ultrawide crop is not really that much different/harder than designing an ultrawide FF lens. The lenses on small P&S cameras have to have focal lengths of like 2 or 3mm on the wide end to get what is considered "wide" for FF, but the lens designs there are not outrageously different than those you find on FF lenses. If you look at the ultrawide crop/FF lens designs, the optical formulae are not significantly altered, rather it's just mostly scaled down.
Full frame does also impact on DoF, but you generally have to be pixel-peeping to see it. The other big difference is down to the quality output - bigger, full frame, sensors have bigger photosites, which in turn leads to higher quality. Cramming 18MP onto a 4/3rds sensor is going to lead to various issues - Canon use things like microlenses to try and improve matters in this area, but the simple fact is that the crop cameras have inferior image quality to their full frame brethren.
You're making a lot of broad generalizations here, and need to be careful to keep things separate. Pixel pitch will determine per-pixel noise level, but there is no universal law stating that FF will have greater pixel pitch than crop sensors. Also when consiering video it gets more complicated because a DSLR sensor does not utilize 100% of its sensor pixels to shoot video. IIRC it "samples" 1920x1080 pixels from the full array and uses that for video. The sensor is not fast enough to sample every point of it's 21MP array 60 times a second and interpolate/downsample to 1080p video. Therefore you could design a crop sensor with only 1920x1080 pixels and have a much bigger pixel pitch than a FF VDSLR and it will perform better.
Lenses are not really any bigger from my knowledge, but then I use Canon's L-series which are not the smallest.
At the wider end of things there can be quite a large difference. The Nikon 17-55 f2.8 vs. the 24-70 f2.8 is a good example (both have a very similar FOV coverage for crop/FF). As you get more towards the telephoto end the difference is not the dominating factor, which is why nobody makes long fast DX primes (i.e. no 400mm f2.8 DX because it would not be much smaller/cheaper than the FX version).
