As batting1000 said, you got your answer. You need to contact Sprint. If you don't want to drive an hour or whatever to get it taken care of, that's fine...it just depends on how much you want your iPhone to be usable.
If the number on the phone is still active after you bought it, there is a slight possibility (please note: this does not translate to me saying "OMG YOU STOLE A PHONE!!1!!1") that the phone you bought from "this one dude who [you] never see around anymore" was stolen.
If this phone belonged to the guy you bought it from, why wouldn't he change his number to a new device? It does not seem likely. There is a slight possibility (see note above) that the phone was obtained through illegitimate means.
But you need to contact Sprint if you want to deactivate the phone and put your number on it. To do this though, you might need the previous account holder's information as I believe you cannot just deactivate a phone from a line without consent of the previous owner.