Most everywhere else, you'll get coverage when in a city or major train line. They have a pretty big 3G wireless infrastructure.
If you goto any of the big parks or countryside, you may loose coverage.
Get one of the International Data plans, you use more data than you think. Maybe 20MB or 50MB per week, depending on how much you think you'll use it.
During a recent trip to Japan, I did about 50MB a week being semi-careful.
Also, I had problems getting data on Softbank in some parts of Japan. I had to manually switch my carrier to NTT DOCOMO to get data. Since Softbank is Apple's carrier in japan, I'm guessing that it's a problem between Softbank and AT&T and not a general softbank issue.