Skip to content
UPEEC

Scraping Libraries

Tools for parsing, crawling and browser automation.

Easy
Comments 0

This is an overview — pick a sub-topic below.

Sub-topics (6)

lxml, Parsel and html5lib

Fast HTML/XML parsing with XPath (lxml), Scrapy's selector library (Parsel), and a browser-exact parser (html5lib).

Easy♥ 0🔖 0💬 0

Scrapy

A full crawling framework: spiders, pipelines, throttling, retries and exports.

Intermediate♥ 0🔖 0💬 0

Playwright and Selenium

Control real browsers to load JavaScript pages, click and fill forms.

Intermediate♥ 0🔖 0💬 0

Pyppeteer, Splash, MechanicalSoup and Newspaper4k

Other tools: Puppeteer-style control (Pyppeteer), a rendering service (Splash), form-filling browser emulation (MechanicalSoup), news-article extraction (Newspaper4k).

Intermediate♥ 0🔖 0💬 0

Trafilatura and Crawl4AI

Extract clean main text from web pages (Trafilatura) and crawl pages into LLM-friendly Markdown (Crawl4AI).

Intermediate♥ 0🔖 0💬 0

Spotted a mistake or want to add detail? Suggest an improvement — Content Contributors' edits are reviewed by editors before they go live.

Discussion (0)

Sign in or join free to ask questions and discuss this topic.