Gat an iPod Touch - you'll be able to do most of what the iPhone can do, but won't have to pay a monthly bill.
Get a crappy phone on a pay as you go sim if you want to call people.
Or, get the iPod Touch and download an app called TextFree (there are other ones too, Skype is pretty popular though I don't know much about Skype) and just pay per minute and have a phone that you can call people as long as you have wi-fi access. No, it won't be as good as a cellphone, but people managed to live without cellphones for a long time (and they wouldn't even be able to use their phone when they were away from home, least it's usable long as you have wi-fi).
And if you are not going to play games, I doubt you'll care about the speed difference. I have a 4 and I do complain it could be faster, but really when I notice the speed difference it is for games. For non game stuff the 4 is pretty decent honestly.
So if you don't want Siri, and you don't play games, and you don't need the best camera (The 4 has a pretty decent camera), get the 4 and save a little money.
(Or do what I just suggested and save a ton of money not paying for a phone plan

).
----------
I think most folks on here gave great advice.
I'm in a similar boat as you. I'm looking to upgrade my 3G and i'm not exactly sure which direction to go in. I could have upgraded my phone back when the iPhone 4 was released but I wasn't that impressed and my 3G was working just fine.
I have no idea how you didn't think the 4 was a significant upgrade to the 3G. I went from the 3G to the 4 and while I liked my 3G even without using the 4 and having it on some version of iOS 3 (I never put iOS 4 on it) I thought it was getting pretty laggy (and not for games, for everything including my grocery list app).
The 4 in speed alone was highly worth the upgrade. It works a helluva lot smoother than the 3G (I really could not go back and not be constantly bitching about the 3G phone after having the 4. I suppose it would be better than a dumb phone but, ugh!). Shoot, even a 3GS would be better (cause I hear the 4 doesn't have that big a leap on speed on the 3GS).
Not to mention the 3G's camera was really bad (when I got it I didn't care cause I didn't use the camera on my phone. But I started using it more and I really would notice how crappy it is). The 4 is a much much better improvement (on everything, including how fast the shutter is. The 3G's shutter had a ton of lag).
The screen is nicer (but that's just a nice extra imho).
Yeah, I really don't see how you thought the 4 was not a significant upgrade over the 3G. In hindsight after seeing how much smoother it ran, even the 3GS for the speed difference alone was a decent upgrade over the 3G (note the 3GS will run iOS 5 fine while the 3G they had to stop supporting iOS 4 period less than halfway through the year even though it barely supported many features of iOS4 and people were warning people not to even put iOS 4 on their 3G cause it would severely slow it down).