FullmetalZ26 has the right idea, but I think his application is a bit flawed.

In his example, you're actually throttled by your internet connection and such a gain is only done if you can somehow link each ethernet connection to a different 'net connect (such as using two cable modems or a cable modem and a DSL modem).
The Mac Pro's dual ethernet ports are a great way to manage network traffic, assuredly. The most common example I know of using dual ethernet ports is to connect a single machine to two different networks.
For example ... a "standard" network, for internet and email access, and a "backup" network for doing networked backups of systems on a regular/constant basis. With Time Machine coming in 10.5, this may be even more common.
Also, a "standard" network and a "rendering" network, where each Mac Pro spends its idle time with Xgrid assignments (as a render node, for example) and handles all of that on the "rendering" network, thus separating that traffic from the "standard" network.
Finally, there's a true "server" application where one ethernet port connects to your broadband connection (cable, DSL, T1, etc.) and the other connects to your LAN, where the server acts as a proxy/firewall/etc.
The same things can be done w/ the iMac, Mac mini, MacBook, or MacBook Pro, using 802.11 for the "standard" network and the faster wired ethernet connections for the backup or rendering connections.
The full range of uses is pretty varied, but I think you get the idea.
EDIT: Also, note that it is giga
bit, not giga
byte.