I had the same issue, however it only specifically affected my MobileMe email. If I quickly took MobileMe mail account offline in Mail.app as soon as it opened then it would remain open and not crash, as soon as I took MobileMe online again it would crash.
Removing the GrowlMail app from the Growl preference pane as well as the growl bundle from the two Library/Mail locations seems to have solved my issue. I never really used growlmail anyway (it would randomly not work anyway!!) so it's no great loss for me.
Interesting bug though, but this is the idea behind public Betas, none of us can complain at all, any other way would risk this bug not being discovered until it was formally released and then people would moan!
Thanks for the suggestions guys, you've solved my issues and I'm happily enjoying Safari 4...so far I'm very very impressed!