https://georgheiler.com/tags/data-orchestration/Data-OrchestrationHugoBlox Kit (https://hugoblox.com)en-usThu, 16 Jul 2026 14:35:00 +0000https://georgheiler.com/media/icon_hu_2b4e40339850646.pnghttps://georgheiler.com/tags/data-orchestration/https://georgheiler.com/event/dagster-slurm-bringing-modern-data-orchestration-to-slurm-managed/Thu, 16 Jul 2026 14:35:00 +0000https://georgheiler.com/event/dagster-slurm-bringing-modern-data-orchestration-to-slurm-managed/<h2 id="abstract">Abstract</h2> <p>dagster-slurm is an open-source Python integration that allows data scientists and research software engineers to run Dagster pipeline assets on both a laptop and Slurm-managed HPC supercomputers without making any code changes. It automatically handles SSH transport, environment packaging via pixi-pack, and Slurm job submission, while streaming logs and scheduler metrics back to the Dagster UI in real time.</p> <p>The talk covers the full workflow, from local development to staging and production deployment on a real HPC cluster, using a live demo with a self-contained Docker Compose environment. It has been validated on VSC-5 in Austria and CINECA Leonardo in Italy.</p> <h2 id="links">Links</h2> <ul> <li>Slides: </li> <li>Session: </li> <li>Project: </li> </ul>