No, you calibrate when you first plug in a new piece of equipment or if you changed the bulb on your projector or change the lighting or location of your projector/monitor.
You want all your sources to match the calibration source.
Most consumer units allow calibration of black levels (very important),
color temp, contrast, brightness and the pro units allow you even deeper access (so you can really screw up your unit

Then when you know that everything is equal you can now see the true differences.
As far as proving which one is better, the same model BR disc player may not match its twin, so until you calibrate, you just don't know. I would say by the pics that the ATV has a factory calibration that is a little more pleasing but less accurate (soft).
I have both and a great transfer like the IMAX photography in Batman
to BR is stunning,
but it's not half bad on the ATV, still breathtaking.
There is no doubt the BR has a better spec then the ATV,
but that will change too in the near future, IMHO.