Dagster-slurm: Bringing Modern Data Orchestration to Slurm-Managed
dagster-slurm is an open-source Python integration that allows data scientists and research software engineers to run Dagster pipeline assets on both a laptop and Slurm-managed HPC …
Incremental Multimodal Graphs
A small local example combining Polars, LanceDB, Metaxy metadata, Lance Graph Cypher, DuckPGQ, and LadybugDB.
Scaling Data Pipelines @Magenta Telekom
Making humans and machines collaborate efficiently.
Upskilling data engineers
A comprehensive guide to modern data engineering with local-first development practices
Local data stack template
Jumpstart your data processing with this local modern data stack template
Cost efficient alternative to databricks lock-in
Spark-based data PaaS solutions are convenient. But they come with their own set of challenges such as a high vendor lock-in and obscured costs. We show how to use a dedicated …
Cloud arbitrage for spark pipelines
Spark-based data PaaS solutions are convenient. But they come with their own set of challenges such as a high vendor lock-in and obscured costs. We show how to use a dedicated …
Cost efficient alternative to databricks lock-in
Save money 💰 and increase developer productivity 👩💻👨💻 by limiting scope-creep of Spark-based data PaaS solutions: 🌐 turn them into an implementation detail 🔧.
Dagster, dbt, duckdb as new local MDS
Lean and efficient MDS experience: Delivers better software engineering practices to the data ecosystem with the new local MDS stack comprised of Dagster, dbt and DuckDB which …
Unlocking Advanced Metadata Extraction with the New DBT API in Dagster
📊 Unleash the power of metadata extraction in your data engineering pipelines with the new DBT API in Dagster! 🚀 Learn how to seamlessly integrate and leverage DBT transformations, …