The 500GB model has two platters with 250GB a piece. Assuming the 320GB unit has two platters at 160GB a piece, the 500GB model will have higher areal density and that should be capable of sustaining higher read and write speeds than the 320GB model.
I have the 500GB model, myself.