Large Mailer Script - Scalability?

[Aperture]

I'm going to be designing a script to check a MySQL database nightly (via a cronjob) to find users who meet certain criteria. It will then need to notify that person via a short email.

This is going to be a fairly large system and it will need to probably email around ~1000ish people an night. (Disclaimer: NO, this is not spam and is on an opt-in basis)

Have any of you accomplished a mailer script on a large scale? What do I need to focus on to ensure scalability and reliability?

Oh, and I'm planning to write this in PHP.

 

[sl1200mk2]

I work in the hosting industry, so hopefully my advice is helpful. In my mind, it's not really the script or scripting per se that's the issue, it's the MTA (mail server) the script is injecting to that's your real concern. This is where the bulk of your reliability and scalability will come from.

If you're running this on a LAMP stack the MTA is likely Sendmail. Sendmail (most modern MTAs for that matter) can easily accommodate 1k emails, however, depending on the environment (shared hosting, VPS, dedicated server, etc) and the resources available you may want to throttle the number of messages introduced to the mail server in batches. As a hosting company, we typically like to see 250 messages ~ 30-60 seconds. Tossing 1,000 messages at once into an already busy mail server can introduce quite a bit of load and impact other users. If this is a VPS or dedicated server it shouldn't be a problem at all, but I would still advise some level of throttling.

If you follow that model and keep the list 'clean' you shouldn't have too many problems. By clean I mean regularly checking / processing bounce messages and removing the dead and/or non-functioning email addresses. Anything less than 10,000 messages really isn't anything your average mail server will have a hard time handling, but again, your environment and likely hosting company will have guidance and limits on the amounts of SMTP traffic you'll be allowed to process in a given period. Seek their advice.

Good luck -

 

[Aperture]

I work in the hosting industry, so hopefully my advice is helpful. In my mind, it's not really the script or scripting per se that's the issue, it's the MTA (mail server) the script is injecting to that's your real concern. This is where the bulk of your reliability and scalability will come from.

If you're running this on a LAMP stack the MTA is likely Sendmail. Sendmail (most modern MTAs for that matter) can easily accommodate 1k emails, however, depending on the environment (shared hosting, VPS, dedicated server, etc) and the resources available you may want to throttle the number of messages introduced to the mail server in batches. As a hosting company, we typically like to see 250 messages ~ 30-60 seconds. Tossing 1,000 messages at once into an already busy mail server can introduce quite a bit of load and impact other users. If this is a VPS or dedicated server it shouldn't be a problem at all, but I would still advise some level of throttling.

If you follow that model and keep the list 'clean' you shouldn't have too many problems. By clean I mean regularly checking / processing bounce messages and removing the dead and/or non-functioning email addresses. Anything less than 10,000 messages really isn't anything your average mail server will have a hard time handling, but again, your environment and likely hosting company will have guidance and limits on the amounts of SMTP traffic you'll be allowed to process in a given period. Seek their advice.

Good luck -

Thanks a bunch, that really helps. This is going to be ran on a dedicated web server, and the mail server on it is not in use... So I don't have to worry about it impacting other users. Thanks again.