Too big is when it starts feeling sluggish. The actual size at which that happens depends on a number of factors, like the speed of the computer, the amount of RAM you have, and the size of each photo. It's a personal decision. If iPhoto starts feeling like it's not enough, switch to Aperture.
jW