I had a similar problem my my Mac Mini where the fans would spin high after very little activity, i've pretty much solved it with 2 things.
1) Replaced the 2x1GB ram sticks with a single 2GB stick which reduced the fan activity by quite a bit, but not totally (don't think this would help your problem though)
2) Installed the latest beta version of FanControl and adjusted the settings so that the fan speed is as low aspossible and the threshold temperatures are as high as possible. I had tried this in the past but the latest beta seems very aggresive in controlling the fan. I can still hear it start to spin up, but after about 3-4 seconds FanControl kicks in and slows it back down to default.