screen scraping data from web database

[chill.]

i play a game called league of legends. it has an online ladder as shown here:

http://www.leagueoflegends.com/ladders/solo-5x5

i'd like to create an application that can perform an automated lookup of an arbitrary player's rating and rank. however, i am having trouble finding a way to access that information.

when i put in a player's name, say "zero2", the page redirects to the ladder page in which zero2 is on. looking through the page source, i can't find a url query for a player name, so i am guessing that the job is done server side.

is there a way for me to simulate a player lookup?

thanks

 

[Darth.Titan]

Well the search form on the page you linked to submits to

/ladders/solo-5x5?35&page=1

and the search text field has a name of "player" so you could try

http://www.leagueoflegends.com/ladders/solo-5x5?35&page=1&player=zero2

However the form is set to submit via "post" so the page may not accept the "player" value if it's provided by a "get" variable in the URL.

You might get lucky though.

 

[X1Lightning]

what kind of application?

web app??(php, jsp, ...)
desktop app?
or phone app?

 

[chill.]

darth.titan, i think your link is the page after you have already searched for "zero2", but your post is helpful and has given me direction in terms of what i should be googling. i found this post http://bytes.com/topic/php/answers/700009-screen-scrape-post-form-results-pages which describes what i am trying to do pretty well, and am working on how to implement now

x1lightning: i'm planning on a desktop app using c#.net and possibly incorporating php and javascript if i need it. but the main problem i have is accessing the data itself that i need for screen scraping

 

[Darth.Titan]

No, the second URL was just an example search link I constructed using the search term you provided. It seems to do the same thing as typing in 'zero2' on the site you linked.

Just take the url "http://www.leagueoflegends.com/ladders/solo-5x5?35&page=1&" and add "player=whatever" to the end.

Might work, might not but it seemed to for me.

Edit: Never mind, apparently that site wants you to see an ad before it will send you onto the link you clicked. That's going to foul any attempt to pass a query string to the search.

 

[HomeBru Studios]

cURL the page then parse the returned HTML, grabbing the bits you need.

Warning: MANY sites have badly formed/non-compliant HTML which makes full/complete parsing difficult - probably best to target just the sections you need, grab it and get out!

 

[chill.]

ok, i think i was able to solve the problem. i used fiddler to determine the POST variables that were necessary, and had to account for the URL redirect which occurred when the POST request was sent. thanks all for the help