There are a few different possible causes for that kind of thing.
A major source of slowdown is just being in an overloaded network area; if you have a bunch of neighbors maxing out their connections, yours will suffer. If you're the only one in your neighborhood with DSL or you do your surfing at 5am, you'll probably get the rated max. If it's time-dependent, this is most likely your issue.
Another possibility is that you've just got a crappy connection; I've certainly seen this happen, both for people who are at the edge of the service area or just have bad wiring in their house. This would be more random, or the same regardless of time of day.
The third cause is if you know that your modem can supply a good speed but you're not getting it on a particular computer. Most frequently this is caused by DNS delays; there have been plenty of threads here about it, and there are several possible solutions. Off the top of my head, I believe "hardwiring" DNS servers in the Network system pref was the most likely to help, but other people had success with things like disabling IPv6 or tweaking other settings.
Personally, I use at&t whatever, too, and I get pretty much the full 6Mbit down/768Kbit up that I pay for, but there are other people in the same neighborhood I know who have had problems with very long DNS delays that are similar to what you're describing; they were apparently caused by their firewall caching an old DNS server, so the computer was having to try several times to get the address. Then there are folks like my brother, who's on Qwest DSL in Denver, but thanks to a bunch of college students and bad wiring in his house, his speeds are miserable and about one day out of every three it doesn't work at all.
Some simple diagnostics: Open up Network Utility and try doing a ping of a reliable site like apple.com. You SHOULD see 0% packet loss, and relatively consistent times in the general vicinity of 100 ms or lower (I see about 30, for example).
If you're getting packet loss, that's a sign of a bad connection, and you're probably going to have to do something about the wiring either in your house or between the wall and the modem. Very long and/or inconsistent times may also be a sign of a bad connection, although I'd think that would more likely be for the entire network rather than your particular corner of it.
Also: Does, say, Apple.com load rapidly for you? If so, then maybe it's just that particular site.
