My iMac (2011, base 21") has been randomly freezing lately, noticeably after upgrading to Lion. I did a hardware test (with the App CD, holding down the D button) and it found no issues.
"i5 and i7 Quad Core iMac computers come with both top memory slots populated. These computers will not start up if only a single DIMM is installed in any bottom slot; these computers should operate normally with a single DIMM installed in any top slot. Core Duo iMac computers should operate normally with a single DIMM installed in any slot, top or bottom. ("Top" and "bottom" slots refer to the orientation of the slots in the pictures below. "Top" refers to the slots closest to the display; "bottom" refers to the slots closest to the stand)."
It appears your iMac will be fine if you switch them around with the top slots only having one RAM stick. It's unclear if additionally having both bottom slots populated might present problems. Find out!