You should see a noticeable difference going from 2GB to 6GB on that machine. I too have an early 2008 24" 2.8GHz machine that came with 2GB; upgrading to 4 GB has kept it fast enough until recently. The update to Lion coupled with Aperture has forced me to run Aperture in 32 bit mode; otherwise, I get spinning beach balls and lots of page outs. So I'll be upgrading mine to 6GB soon.
The primary reason for upgrading your HDD would be if you are running out of space; that will slow it down. I believe that a general rule of thumb is that you should try to keep 15% free; so if you have a 320GB drive you should try to keep ~50GB free. I upgraded my 320GB to a 1TB drive less than a year ago for that very reason (plus I wanted a larger drive) I believe the stock 320GB drive already was 7200 RPM, so it was a fairly fast drive to begin with.
The next step would be to go to an SSD. You mainly get faster boot times, faster program loads, and less latency if/when you have page outs (which is why the Macbook Air is so fast but even with only 2GB RAM - you don't notice page outs). I am considering adding an SSD to my machine as a primary boot/app drive, but there are some issues with doing that as well. I'd prefer to also have a large HDD for files and such, which means removing the optical drive and putting a HDD in that bay. Unlike the newer iMacs, the optical drive in the 2008 has a PATA interface, so you can't just replace your optical drive with an SSD and get the speeds you want. Best thing to do performance-wise is to replace your HDD (which is on an SATA bus) with an SSD, then put a 2.5" HDD in a PATA-to-SATA adapter, and put that whole assembly into the optical drive bay. That means buying yet ANOTHER HDD, as the internal drives in the iMac are 3.5". However, I have two external enclosures that could use a bigger drive in them, so the 1TB would not go to waste if I went ahead and did this update.