It gets complicated.
There are two ways to send an iMessage: as a telephone number message, or as an Apple ID message. In order for your messages to stay in sync between multiple devices you must use an Apple ID exclusively, and it must be set up in the Messages settings pane of both devices. Also, ensure the "Caller ID" setting in Messages settings is the same Apple ID on both devices.
You can still send an iMessage to a phone number when you have it set up as an Apple ID, and then people will reply to your Apple ID. Sending SMS messages will work as they always have, and will obviously not appear on iPad.
One caveat: any existing text message conversations from iOS 4 are considered phone number-to-phone number, and will never sync with iPad. Even if it automatically converts the conversation to an iMessage conversation, it's still phone number-to-phone number. You'll need to start the conversations again as a new message, once you have your Caller ID set correctly as described above. Then everything will stay in sync.
In short, you need to use iMessage with an Apple ID exclusively to stay in sync. If you receive an iMessage to your phone number, start a new thread to that person so you're sure you're using your Apple ID.
Eventually, people will be converted to using Apple IDs for iMessages and all of this will be behind us.