How to Make a Web Crawler With Selenium

Web Crawling is extremely useful to automate certain tasks performed routinely on websites. You can write a crawler to interact with a website just as a human would do. In an earlier article, we covered the basics of writing a webcrawler using the python module, scrapy. The limitation of that approach is that the crawler does not support javascript. It will not work properly with those websites that make heavy use of javascript to manage the user interface. For such situations, you can write a crawler which uses Google Chrome and hence can handle javascript just like a normal user-driven…

Read the full article: How to Make a Web Crawler With Selenium

MakeUseOf