Demonstration of an end-to-end data station with Dagster as the main orchestrator. This data station follows the solution design of a lakehouse, leveraging new technologies such as Polars and DuckDB such that it can process up to 1 TB of data on a single node. Data stations are envisaged to parcipitate in a network for federated analytics or in a data mesh.

Option 1: uv

Ensure uv is installed following their official documentation.

Create a virtual environment, and install the required dependencies using sync:

uv sync

Then, activate the virtual environment:

Option 2: pip

Install the python dependencies with pip:

python3 -m venv .venv

Then activate the virtual environment:

Install the required dependencies:

pip install -e ".[dev]"

Start the Dagster UI web server:

dg dev

Open http://localhost:3000 in your browser to see the project.