PSA: Hit a wall on my first web scraper project last weekend

I spent 3 hours last Saturday trying to scrape data from a local news site and kept getting blocked by their robots.txt file. Turned out I needed to add a user-agent string and slow down my request rate to 2 seconds between each page. Has anyone else run into this kind of issue when starting out with Python scraping?

0 comments

0 Comments

No comments yet

Be the first to share your thoughts on this discussion.