What exactly is infuriating about it? I haven't used either very much. I have a MacBook pro, and I like the trackpad on that compared to other laptops, but I'd still rather use a mouse.
I find it (and the Mighty Mouse) too low-profile and prefer a chunkier ergonomic mouse. The gestures don't work for me, and if you collide with something on your desk it registers a click.
Thing is, you really need to choose a mouse that feels comfortable in your hand - the Magic Mouse might work for some people but it only comes in one shape and size.
I've got a Logitech Performance MX. Nice chunky mouse with a proper wheel. Tiny USB transmitter that you can leave plugged into a laptop (mine's mostly plugged into the USB through port on an Apple keyboard) and a sensibly designed micro-USB charging socket that lets you keep using the mouse while it is re-charging.
I bought a Magic Trackpad out of curiosity - its great for basic 'productivity' use - even nicer than the MacBook trackpad, works nicely with the gestures in Lion. Enabling the three-fingers-to-drag option helps, too. However, I still prefer the mouse for graphics work and gaming.
Mice and keyboards have always been Apple's achillies heel - for years I want for Microsoft or Logitech instead. I was finally won over by the Apple aluminium chicklet keyboard, but they still haven't nailed the mouse. Ironic, for the company that popularised mice in the first place...