Web page Bandwith

Discussion in 'Web Design and Development' started by Halcyon, Jul 24, 2007.

  1. Halcyon macrumors 6502

    Joined:
    Sep 21, 2006
    #1
    Problem here.
    I have a web page that is constantly exceeding its alloted monthly bandwith...unfortunately not because of welcomed visitors but from spammers and probably boots.

    I have checked the statistics for the page in question and cannot really pinpoint one single IP responsible for this. I was thinking that I could block this IP but it changes constantly. The only trend I've been able to discern is that the offender will navigate to every single page in the site and will "inspect" every object that can be clicked or view (links, pictures, etc.). Being that no real person does this, it has to be some kind of robot doing it...with what purpouse? I don't know. It even "visits" linked css style sheets and PHP include files.

    Any ideas or suggestions as to how to deal with this?

    TIA
     
  2. RojoLeo macrumors 6502

    Joined:
    Mar 11, 2007
    Location:
    Austin, TX
  3. Halcyon thread starter macrumors 6502

    Joined:
    Sep 21, 2006
    #3
    Nope, no robot.txt file yet. What I have done instead is place a meta tag that should work equally well, even though not all robots implement it yet:

    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

    I'll set up a robot.txt fil, implement it and see if it helps.

    Thanks.
     
  4. angelneo macrumors 68000

    Joined:
    Jun 13, 2004
    Location:
    afk
    #4
    What kind of site are you running? I don't think any indexing bots can used up that much bandwidth, it could be spammers though. If it is really spammers, I don't think they will honor your robots.txt file. How do you gather that spammers' visits are disabling your site?
     
  5. Halcyon thread starter macrumors 6502

    Joined:
    Sep 21, 2006
    #5
    I simply don't know. Just like you mention, I don't think indexing robots will use all that bandwith (at least I've never seen or heard of that happening) so I'm assuming maybe spammers are the offenders. Then again, this particular web site has no content whatsoever that can be of interest to spammers (it's a small business/factory web site and the contents is text and some images).

    I've been monitorng traffic for the past 24 hours and it seems to have returned to "normal" levels after reaching a peak of 10 to 20 times it normal high during the past weekend. I'll keep monitoring and hope for the best.

    Thanks for our help.

    PD - Other people I've consulted with agree with you in that robots.txt files will not stop spammers (at least not the ones that are good at it)...and that most probably it was indeed some sort of spamming that generated this. Once the spammer found nothing of interest (info, reaction, whatever...) it just moved on to another poor soul like myself :)
     
  6. zimv20 macrumors 601

    zimv20

    Joined:
    Jul 18, 2002
    Location:
    toronto
    #6
    is there a stats package bundled with your hosting? have you looked through it to see what files are being requested?
     
  7. Halcyon thread starter macrumors 6502

    Joined:
    Sep 21, 2006
    #7
    Yes. Webalizer is the name of the package and I also run StatCounter which I find far superior for other info.

    I've checked the logs and like I mentioned before they go after everything...all files, links, pictures, PHP include files, css style sheets, etc.
     
  8. zimv20 macrumors 601

    zimv20

    Joined:
    Jul 18, 2002
    Location:
    toronto
    #8
    do you have that little bandwidth available, or are they grabbing a whole bunch?

    i've got a bunch of stuff in my .htaccess that cuts down on fraudulent signups on a joomla site. if you don't mind blocking the entire .ru population, and stuff like that.

    for example:
    Code:
    deny from .vn
    deny from .gr 
    deny from .ru
    deny from .kr
    deny from .ar 
    deny from .bz
    deny from .sg
    
    deny from megagiga.com
    deny from netzero.net 
    deny from spaceproxy.com
    deny from anonymizer.com
    deny from drikka.net
    deny from netdirect.net
    deny from juno.com
    deny from voyager.net
    deny from globali.net
    deny from the-cloak.com
    deny from anonymouse.ws
    deny from megaproxy.com
    
     
  9. angelneo macrumors 68000

    Joined:
    Jun 13, 2004
    Location:
    afk
    #9
    Where did you get this list? Is it based on your experience? I'm kind of surprised to see .sg being blocked from your site.
     
  10. zimv20 macrumors 601

    zimv20

    Joined:
    Jul 18, 2002
    Location:
    toronto
    #10
    i gathered bits and pieces from the joomla security forums. i didn't generate the list based on my experience, but my experience in using it (and some other rules) was a dramatic reduction (90-95%) of fraudulent signups.

    that site is for a small, chicago-based theater company, and i'm willing to pay the price of losing our singapore audience to keep my webmastering sanity.

    fwiw, i did add the countries one or two at a time. surprisingly, denying .ru didn't cut down that much trouble, but adding .vn did.
     
  11. angelneo macrumors 68000

    Joined:
    Jun 13, 2004
    Location:
    afk
    #11
    Is there any way to convince you to lose the .sg rule? :p:). (you can ignore this btw)

    Anyway, to the OP, good luck to maintaining the traffic, let us know if the problems come back.
     
  12. zimv20 macrumors 601

    zimv20

    Joined:
    Jul 18, 2002
    Location:
    toronto
    #12
    i think it's worth a shot: why should i restore .sg?
     
  13. angelneo macrumors 68000

    Joined:
    Jun 13, 2004
    Location:
    afk
    #13
    Just a suggestion, because I come from Singapore myself, the laws here are pretty strict, you probably have heard of the ridiculous no chewing gum rule, and the caning punishment. Recently, we just got a teenager prosecuted for leeching WIFI.... If you managed to get the IP and ISP of the spammer who originated from .sg, chances of you getting some response from the agency here are pretty high if it is really serious. Oh ya, also, they just passed a new anti-spam law this year as well.

    Anyway, I will still respect your decision even if you still keep that rule. :)
     
  14. zimv20 macrumors 601

    zimv20

    Joined:
    Jul 18, 2002
    Location:
    toronto
    #14
    nah, you convinced me. off it comes!
     
  15. angelneo macrumors 68000

    Joined:
    Jun 13, 2004
    Location:
    afk
    #15
    Thanks :)
     
  16. zimv20 macrumors 601

    zimv20

    Joined:
    Jul 18, 2002
    Location:
    toronto
    #16
    aaaaand it's back on the deny list. my fraudulent signups went up over 1000% during that time.
     
  17. angelneo macrumors 68000

    Joined:
    Jun 13, 2004
    Location:
    afk
    #17
    Woah, well, sorry for the trouble then.
     

Share This Page