Answers to a few questions might help...
Downloading via the Web, via FTP, via SCP, ???
What programs are you using to do the downloading (inside of VPC, and in Panther)?
Did you have the VPC session running when you tested the Panther download speed?