jade_grant95•1mo agoMost Upvoted

PSA: Hit a wall on my first web scraper project last weekend

I spent 3 hours last Saturday trying to scrape data from a local news site and kept getting blocked by their robots.txt file. Turned out I needed to add a user-agent string and slow down my request rate to 2 seconds between each page. Has anyone else run into this kind of issue when starting out with Python scraping?

2 comments

2 Comments

david_stone1mo ago

Start by actually reading the robots.txt file before you complain about it. Most of the time those rules are for bots and crawlers, not a regular scraper running from your laptop. I've hit the same wall but found that adding a simple user-agent and waiting 3-5 seconds between requests fixed it every time. You don't need a 2 second delay, that's too fast for most sites and they'll still block you. Try 5 seconds and see if that works better for the local news site. Also check if they have any rate limiting in their header responses, that tells you exactly what they allow.

the_jennifer1mo ago

Have you tried setting the user-agent to something that looks like a real browser? I had the same issue with a local site and adding a Firefox user-agent string plus a 4 second delay did the trick for me...