Scraping Concepts

Techniques and responsibilities.

Easy

This is an overview — pick a sub-topic below.

Sub-topics (4)

Locate data in a page's structure using CSS selectors (`div.price`) or XPath (`//div[@class='price']`).

Easy♥ 0🔖 0💬 0

Static pages contain the data in the HTML; dynamic pages load it with JavaScript.

Intermediate♥ 0🔖 0💬 0

Follow links and "next page" buttons systematically while pacing requests.

Intermediate♥ 0🔖 0💬 0

robots.txt tells crawlers which paths to avoid; ethics and law go further.

Easy♥ 0🔖 0💬 0

Spotted a mistake or want to add detail? Suggest an improvement — Content Contributors' edits are reviewed by editors before they go live.