Mine was acting sluggish but after I turned it off and then back on it was fine.
Do you mean turning off/on the iphone or the contacts app? If the latter one how do you do that?
I had the same thing yesterday, it wwould show my contacts but freeze for several seconds. I restored to build 347 and it's now instant.
Likewise, doing a restore and updating the firmware made no difference in the contact list or SMS typing speed. Both are still awfully slow at times. It's not hard to repro the problem, I'd assume (right now) that Apple will find and fix whatever the problem happens to be. I have maybe 30 contacts so that can't be the problem.