Mark As Completed Discussion

2. Web scraper (crawler)

A web scraper or a crawler is an automated script designed to surf the internet and store the content of certain web pages. A web crawler is especially useful to find up-to-date information using multi-thread concepts in its program. You can easily build a crawler bot using Python’s request module or Scrapy, Python’s open-source web crawling framework explicitly designed for web scraping and extracting data by using APIs. You can write scripts for gathering specific information and then store it in a CSV or an Excel file.

Building a web scraper with Python offers you the opportunity to learn how web crawlers work in real-life applications and make an app or script that will get you some data you might actually need!

On this link, you can find an example of a web crawler in Python, and get an idea about this kind of application.

2. Web scraper (crawler)