At my Family we have an iPod 1G, 2G and 3G. Obviously the 1G is on 3.1.3 because it can't get further but both the 2G and 3G were much better on 3.0 than are now on 4.0, they are much slower and the battery drains faster.
If you really care about the new features such as GameCenter and Folders than you should upgrade, but if you don't then update to 3.1.3, it's a good version and if you ever want to go to 4.2.1 then just connect it to iTunes and update.