Dagster-slurm: Bringing Modern Data Orchestration to Slurm-Managed
McNamara Alumni Center, Minneapolis, Minnesota
Abstract
dagster-slurm is an open-source Python integration that allows data scientists and research software engineers to run Dagster pipeline assets on both a laptop and Slurm-managed HPC supercomputers without making any code changes. It automatically handles SSH transport, environment packaging via pixi-pack, and Slurm job submission, while streaming logs and scheduler metrics back to the Dagster UI in real time.
The talk covers the full workflow, from local development to staging and production deployment on a real HPC cluster, using a live demo with a self-contained Docker Compose environment. It has been validated on VSC-5 in Austria and CINECA Leonardo in Italy.
Links
Researcher at the Supply Chain Intelligence Institute Austria (ASCII).
My research interest lies at the intersection of forecasting extreme events and causal analysis in high-frequency time series.