data-engineering

Apache Superset is a Data Visualization and Data Exploration Platform

Learn how to develop, deploy and iterate on production-grade ML applications.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

Turns Data and AI algorithms into production-ready web applications in no time.

Workflow Engine for Kubernetes

An orchestration platform for the development, production, and observation of data assets.

The Data Engineering Cookbook

Graph-Native Infrastructure for Context and Accountable AI Systems

Roadmap to becoming a data engineer in 2021

Always know what to expect from your data.

Incremental engine for long horizon agents 🌟 Star if you like it!

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

🧙 Build, run, and manage data pipelines for integrating and transforming data.

Fancy stream processing made operationally mundane