Processing Engines

Spark, Flink, Beam, Kafka and DuckDB.

Intermediate

This is an overview — pick a sub-topic below.

Sub-topics (8)

Distributed processing of very large datasets with a DataFrame API.

Intermediate♥ 0🔖 0💬 0

Stream processing engines (Flink) and a unified batch/stream programming model (Beam) with Python SDKs.

Advanced♥ 0🔖 0💬 0

Distributed event logs for streaming data; Redpanda is a Kafka-compatible alternative.

Intermediate♥ 0🔖 0💬 0

Python-native stream processing libraries (Faust: Kafka streams; Bytewax: Rust-powered dataflows).

Advanced♥ 0🔖 0💬 0

An in-process analytical SQL database — query CSV/Parquet files and DataFrames at high speed.

Easy♥ 0🔖 0💬 0

One Python DataFrame API that compiles to many backends (DuckDB, Postgres, BigQuery, Spark).

Intermediate♥ 0🔖 0💬 0

Fast loading from databases into DataFrames (ConnectorX), and simple declarative data loading pipelines (dlt).

Intermediate♥ 0🔖 0💬 0

Lightweight ETL libraries (petl, Bonobo) and connector ecosystems (Singer taps, Meltano, Airbyte) for moving data between systems.

Intermediate♥ 0🔖 0💬 0

Spotted a mistake or want to add detail? Suggest an improvement — Content Contributors' edits are reviewed by editors before they go live.