trackpad won't click

I tried searching MRoogle, while I could find many trackpad threads, I couldn't find a trackpad issue that was the same as mine.

Basically, the trackpad works just fine most of the time.

However, sometimes (often enough to be aggravating), it will not click.

A typical scenario is this:
I'll have a web browser or something open and I will want to close the window. So I go to click the red button. The cursor is over the red button, and the button turns to an x, notifying that I am in the right spot. I will click the trackpad, and it will make the click sound, and nothing will happen. I'll try clicking several more times and nothing. The click just won't register no matter how many times I click.

What I will have to do to make it work is stop clicking and take my hand off the trackpad for like 5-6 seconds, THEN click, and it will work.

Is this common? Does anyone know of a fix for this?

I do not have tap to click on. This is a 15" MBP 2.4ghz i5.