Not necessarily, what determines the physical limits is pixel density. So the advantage of full frame sensors is that they either allow for much larger resolution or for lower pixel density which improves the theoretical limits regarding noise behavior and dynamic range.
Generally speaking, full frame sensors do have lower pixel density and thus perform better. However, if you have a full frame camera, you need very good lenses -- and making lenses that perform well on full frame tend to be more expensive than lenses which perform well on crop sensors.
Olympus' pro lenses, for instance, are often more compact than their full frame counter parts and offer stellar optical performance. The downside is that Olympus' cameras cannot compete with modern full frame cameras when it comes to (very) high ISO behavior.
Image quality is determined by the whole optical system and not just by the sensor.