screen scraping data from web database

Discussion in 'Web Design and Development' started by chill., Oct 11, 2010.

  1. chill. macrumors 6502

    Joined:
    Sep 1, 2008
    #1
    i play a game called league of legends. it has an online ladder as shown here:

    http://www.leagueoflegends.com/ladders/solo-5x5

    i'd like to create an application that can perform an automated lookup of an arbitrary player's rating and rank. however, i am having trouble finding a way to access that information.

    when i put in a player's name, say "zero2", the page redirects to the ladder page in which zero2 is on. looking through the page source, i can't find a url query for a player name, so i am guessing that the job is done server side.

    is there a way for me to simulate a player lookup?

    thanks
     
  2. Darth.Titan macrumors 68030

    Darth.Titan

    Joined:
    Oct 31, 2007
    Location:
    Austin, TX
    #2
    Well the search form on the page you linked to submits to
    Code:
    /ladders/solo-5x5?highlight=35&page=1
    and the search text field has a name of "player" so you could try
    Code:
    http://www.leagueoflegends.com/ladders/solo-5x5?highlight=35&page=1&player=zero2
    However the form is set to submit via "post" so the page may not accept the "player" value if it's provided by a "get" variable in the URL.

    You might get lucky though.
     
  3. X1Lightning macrumors 6502

    X1Lightning

    Joined:
    Feb 19, 2007
    #3
    what kind of application?

    web app??(php, jsp, ...)
    desktop app?
    or phone app?
     
  4. chill. thread starter macrumors 6502

    Joined:
    Sep 1, 2008
    #4
    darth.titan, i think your link is the page after you have already searched for "zero2", but your post is helpful and has given me direction in terms of what i should be googling. i found this post http://bytes.com/topic/php/answers/700009-screen-scrape-post-form-results-pages which describes what i am trying to do pretty well, and am working on how to implement now

    x1lightning: i'm planning on a desktop app using c#.net and possibly incorporating php and javascript if i need it. but the main problem i have is accessing the data itself that i need for screen scraping
     
  5. Darth.Titan macrumors 68030

    Darth.Titan

    Joined:
    Oct 31, 2007
    Location:
    Austin, TX
    #5
    No, the second URL was just an example search link I constructed using the search term you provided. It seems to do the same thing as typing in 'zero2' on the site you linked.

    Just take the url "http://www.leagueoflegends.com/ladders/solo-5x5?highlight=35&page=1&" and add "player=whatever" to the end.

    Might work, might not but it seemed to for me.

    Edit: Never mind, apparently that site wants you to see an ad before it will send you onto the link you clicked. That's going to foul any attempt to pass a query string to the search.
     
  6. HomeBru Studios macrumors member

    HomeBru Studios

    Joined:
    Jun 4, 2008
    #6
    cURL the page then parse the returned HTML, grabbing the bits you need.

    Warning: MANY sites have badly formed/non-compliant HTML which makes full/complete parsing difficult - probably best to target just the sections you need, grab it and get out!
     
  7. chill. thread starter macrumors 6502

    Joined:
    Sep 1, 2008
    #7
    ok, i think i was able to solve the problem. i used fiddler to determine the POST variables that were necessary, and had to account for the URL redirect which occurred when the POST request was sent. thanks all for the help
     

Share This Page