If you're having problems, or need to emulate some form POSTs or ajax, get Firefox and install theLiveHttpHeaders plugin. This plugin will allow you to manually browse the site and capture the urls being navigated along with any cookies that are being passed during your manual browsing. That is what you need your scraperbot to send in a request to get a valid response from the target webserver(s). This will also capture any ajax calls being made, and in many cases the same ajax calls must be implementated in your scraper to get your desired responses.
screen scraping get javascript link
http://stackoverflow.com/questions/125177/whats-a-good-tool-to-screen-scrape-with-javascript-support
No comments:
Post a Comment