robots.txt and Ethical Scraping

robots.txt tells crawlers which paths to avoid; ethics and law go further.

Easy

urllib.robotparser reads robots.txt. Respect terms of service, copyright and privacy law; don't overload servers or bypass logins.

Spotted a mistake or want to add detail? Suggest an improvement — Content Contributors' edits are reviewed by editors before they go live.

Discussion (0)