I'm currently trying to access a webpage to gather information on a service to organize into a report. The service is a networking directory for truck loads that need to be picked up and shipped across the continental United States. My project is to query this webpage for the total number of shipments available.
The webpage has a great search tool for querying shipments, and I've been asked to query by Company name (maybe they will eventually have me organize number of loads available by company name). The companies can be found in a selection list called 'company'. I imagine the process of selecting this list, choosing an option, and pressing the 'Search for Loads' button is common knowledge for everyone here, and that's not really my question.
The problem I've found is that once I submit the request, Mechanize automatically pretty prints the page and I can see a list of page links that, when counted, equal the number of shipments available from that company, which is perfect. HOWEVER, no matter what my next command/request is (even if I just pretty print the page again) the search results disappear
At this point, I'm at a loss for what I can do to gather information on the search results. I've tackled the same issue using Watir Webdriver and got it to work really well simply by having the bolded text printed to the screen (the number the page displays as the number of search results is the ONLY bolded text on screen), but I think it'd be annoying to write a script to automate this process if we still have to have a browser running in the background (as Watir does).
If anyone can help give a nudge in the right direction with this, I'd be extremely grateful. The link to the page:
http://www.freight-terminal.com/loadlist
You can get source code from there if you need it. Thanks again for any help!
The webpage has a great search tool for querying shipments, and I've been asked to query by Company name (maybe they will eventually have me organize number of loads available by company name). The companies can be found in a selection list called 'company'. I imagine the process of selecting this list, choosing an option, and pressing the 'Search for Loads' button is common knowledge for everyone here, and that's not really my question.
The problem I've found is that once I submit the request, Mechanize automatically pretty prints the page and I can see a list of page links that, when counted, equal the number of shipments available from that company, which is perfect. HOWEVER, no matter what my next command/request is (even if I just pretty print the page again) the search results disappear
At this point, I'm at a loss for what I can do to gather information on the search results. I've tackled the same issue using Watir Webdriver and got it to work really well simply by having the bolded text printed to the screen (the number the page displays as the number of search results is the ONLY bolded text on screen), but I think it'd be annoying to write a script to automate this process if we still have to have a browser running in the background (as Watir does).If anyone can help give a nudge in the right direction with this, I'd be extremely grateful. The link to the page:
http://www.freight-terminal.com/loadlist
You can get source code from there if you need it. Thanks again for any help!
