I want to set up a website that will scrape a number of website, starting with one but the option to expand into 5+. The first website has around 20,000 pages and I have a sitemap of the pages I want to scape.
I have my PHP code to scrape and do all the bits it needs to do, question is, what is the best implimentation to do this.
I was looking at a amazon EC2 and run the script on there as a cron, but as a cron job it seems rather old and out dated?
any one able to advise on a better method?
I have my PHP code to scrape and do all the bits it needs to do, question is, what is the best implimentation to do this.
I was looking at a amazon EC2 and run the script on there as a cron, but as a cron job it seems rather old and out dated?
any one able to advise on a better method?