Can't Change Apple ID

I'm trying to change my Apple ID to another of my email addresses that was formerly used as a "rescue email" on the same account. This email address is not longer associated with my Apple account. However, when I try to change my Apple ID to this email, I get this error message: "This email address is already in use with another Apple ID." That is not true as it should no longer be associated with my Apple account. How do I enable my account to change the Apple ID to this email address?