Dagster-slurm: Bringing Modern Data Orchestration to Slurm-Managed
dagster-slurm is an open-source Python integration that allows data scientists and research software engineers to run Dagster pipeline assets on both a laptop and Slurm-managed HPC …
Metaxy: Record-Level Feature Metadata Management for Multimodal ML Pipelines
Software that processes large datasets often repeats expensive computations when any part of the input data or processing logic changes. Metaxy is about perfecting the art of doing …
VDSG: Optimizing Multimodal AI Pipelines with Metaxy
The AI era has shifted compute toward complex multimodal pipelines, where a small input or code change can trigger expensive downstream recomputation. This talk introduces Metaxy, …
SAO: Metaxy - Field-Level Metadata Management for Incremental Multimodal ML Pipelines
Multimodal machine learning pipelines that process video, audio, and images incur substantial GPU costs. Existing data orchestrators operate at table or asset granularity: when any …
Perfecting the Art of Doing Nothing: Incremental Multimodal AI Pipelines with Metaxy
AI pipelines are now expensive enough that recomputing more than necessary is the dominant cost. Tokens and GPU hours change the economics, and agentic workflows branch and …
Docling + Metaxy: Patent Intelligence at Scale
A walkthrough at the Docling Community Office Hours of how Jubust combines Docling for document parsing with Metaxy as an incremental, field-level metadata control plane to build …
Discovering the SUPER in computing - dagster-slurm for reproducible research on HPC
Dagster is a modern data orchestrator that emphasises reproducibility, observability, and a strong developer experience. In parallel, most high-performance computing (HPC) centres …
Metaxy + Dagster-Slurm for Efficient Multimodal Pipelines
How to combine Metaxy, Dagster-Slurm, Docling, and Ray to run incremental multimodal pipelines on sovereign AI infrastructure.
Dagster + Slurm
Run and orchestrate Dagster workloads on Slurm-based HPC environments with production-oriented patterns.
•
1 min read
Scaling Data Pipelines @Magenta Telekom
Making humans and machines collaborate efficiently.