I think iOS has a memory clearing issue, which at some point begins to effect all aspects of the iPhone. I have an iPhone 4, 32gb running iOS 5 beta 2. At first, beta 2 was working great, but then it began to slow down and I started having issues with the iPhone's performance, including with the home button.
Fix? Go in a few times a day, and close out all your open programs in the multitasking bar. This seems to alleviate the slow downs, and bugs significantly. Hope this helps!