I have the same problem. I have 1243 photos (with a handful of videos mixed in) that amount to 9.6GB (determined by the "Info" panel in iPhoto). Yet, checking the iPhoto ibrary (user/Pictures/iPhoto Library) shows that iPhoto is taking up about 15GB.
To try to fix this, I saved all my iPhoto photos into an external location, wiped and rebuilt the iPhoto library, then imported back in the photos I had before. This reduced iPhoto's size appropriately to around 9.6GB.
However, recently I imported about 300 photos, which I cut down to around 60, but even after emptying the iPhoto trash and Finder trash, iPhoto added on another couple gigs (it's now back up to 15GB).
Before I rebuilt the iPhoto library, I checked the "iPhoto Library" package and found many photos I'd deleted still in there. My suspicion is that that is still happening.
I'm using iPhoto '11 by the way.