Dagster, Airflow, or Prefect, Lessons From Building a Package Intelligence Pipeline - Part 2
Introduction
In the previous article, I discussed the requirements for a Data Orchestrator for the Data intelligence pipeline that I’m building to read and classify package data from NuGet as an open-source or not. To recap the requirements:
- Data as first-class concept: The orchestrator thinks in datasets with schemas, freshness, and dependencies, not just jobs that happen to produce data.
- Incremental Processing & Partitioning: Each slice of data is tracked, retried, and backfilled independently, so you never reprocess what hasn’t changed.
- Dependency Graphs & Change Detection: The orchestrator knows what depends on what, and re-runs only the assets an upstream change actually affects.
- Event-based Triggers: Materialization fires when new data arrives, not when the clock says so.
- Data Quality Gates: Bad data is caught at the asset that produced it, before it cascades into every downstream consumer.
- Pipeline-level Observability: Orchestrator can provide both granular and holistic observability and helps answer questions like “is my data fresh, correct and on-time?”
There are numerous Data Orchestrators in the industry and having considered the above requirements, I have picked three strong contenders Dagster, Apache Airflow and Prefect.
Why these three?
Temporal and Flyte sit in different categories. Temporal is a workflow and activity engine, not a data orchestrator. Flyte is asset-aware but ML-centric and carries a heavier infra footprint. dbt and SQLMesh are transform-only and assume the data is already in the warehouse; this pipeline starts at ingestion. Airbyte and Fivetran are connectors that live inside a pipeline phase, not orchestrators themselves. In this article, I want to focus on comparing the three major data orchestrators so that I can pick one to move ahead in building the data pipeline.
The three tools
Dagster
Dagster is an open-source Data Orchestrator built around the concept of software-defined data asset. Each data asset is defined declaratively which includes code to produce the data and the dependency information related to the data asset (Upstream and downstream dependencies). Consider the NuGet pipeline that I’m building, I would declare a data asset “NuGet_packages” instead of defining just the job that produces the NuGet package data. Within the “nuget_packages” asset, I point to the function that materializes the data to Dagster. Dagster will take care of building the dependency graph from the asset definitions themselves. Partitions, schemas, freshness policies, and quality checks attach to the asset, not to the task that produced it. The mental shift is from “what runs when” to “what should exist, and is it current?”
Apache Airflow
Apache Airflow’s original abstraction is the task DAG: a directed graph of operators that execute on a schedule. Each task does work, but doesn’t inherently know what data it produced. Data is implicit in Airflow’s original model, a side effect of tasks running in order. Airflow 2.4 introduced Datasets to make data-aware scheduling possible, and Airflow 3 has pushed this further with Assets as a first-class concept, dataset-driven triggers, and a redesigned scheduler. It carries the deepest operational maturity and the largest connector ecosystem of the three, with the asset model layered on top of its task-centric foundations.
Prefect
Prefect’s primary abstraction is the flow: a Python function decorated as a flow, composed of tasks, with state and results tracked by the Prefect engine. Where Airflow leans on the DAG and Dagster on the asset, Prefect leans on the flow as a pythonic unit of work which is dynamic by default, easy to parameterize and easy to develop. Prefect 3 added assets, automations, and transactions, bringing the data-aware model closer to Dagster’s, while keeping the flow rather than the asset at the center.
Comparing against the requirements
| Requirement | Dagster | Apache Airflow 3 | Prefect 3 |
|---|---|---|---|
| Data as a first-class concept | Software-defined assets are the core abstraction. Schemas, metadata, freshness, and lineage attach to the asset itself. | Assets (3.x) and Datasets (2.4+) layered on top of the task DAG. First-class in name, task-centric in heritage. | Flows are the primary citizen; assets added in Prefect 3 but the flow remains the center of gravity. |
| Incremental processing & partitioning | Native partitioned assets. Per-partition materialization, retries, and backfills built in. | logical_date / data interval per DAG run; backfill via CLI. Partitioning is per-run, not per-dataset slice. | No native partition primitive. Done via parameterized flow runs; cursor and state management is on the developer. |
| Dependency graphs & change detection | Asset graph is the dependency graph. Code versions and observable sources drive stale detection; declarative automation re-runs only what’s affected. | Explicit task DAGs; cross-DAG deps via Datasets and sensors. Change detection mostly schedule-driven rather than asset-version-driven. | Task graph within a flow, subflows across flows. DAG is dynamic and resolved at runtime; no built-in stale-asset detection. |
| Event-based triggers | Sensors (asset, run-status, multi-asset) plus declarative auto-materialize / automation policies. | Data-aware scheduling on Assets/Datasets, Asset watchers in 3.x, traditional sensors for externals. | Events + Automations via webhooks, custom events, state-change triggers. Strong runtime eventing model. |
| Data Quality Gates | Asset checks are a first-class primitive attached to the asset itself; failures block downstream materialization. | Not native and Implemented as tasks (SQLCheckOperator, Great Expectations integration, custom operators). | Not native and implemented inside tasks or via result validation. Transactions in Prefect 3 provide rollback semantics. |
| Pipeline-level observability | Asset lineage, materialization history, freshness state, and check status surfaced in one UI model. | Mature task/run UI; data lineage via OpenLineage integration. Asset-level views still maturing. | Run-centric UI with deep flow/task introspection; asset lineage and catalog newer in Prefect 3. |
The verdict
Dagster meets all six requirements with native primitives. Airflow and Prefect either layer assets on top of an older abstraction or leave gaps for me to fill. For the NuGet pipeline specifically, three rows of the table mattered most:
Incremental processing and partitioning
NuGet has millions of packages. A full refresh on every run isn’t feasible, so the only practical model is to slice the work by day and materialize each slice independently. Dagster gives me per-slice retries and backfills out of the box. Airflow’s partitioning is per-run rather than per-dataset slice, and Prefect leaves it to me to build on top of parameterized flow runs.
Data Quality Gates
The pipeline classifies packages by running classify over enriched signals. Bad data at the ingestion or enrichment phase silently corrupts the classification output, and re-materialization is expensive. Dagster’s asset checks attach quality validation to the asset itself and block downstream materialization on failure. In Airflow and Prefect, quality gates are a pattern I implement inside tasks rather than a primitive the orchestrator enforces.
Dependency graphs and change detection
The four phases of the pipeline (Ingest → Enrich → Classify → Index) need to re-run selectively when an upstream signal or classification logic changes, not on every run. Dagster derives the dependency graph from the asset definitions and uses code versions and observable sources to detect what’s stale. Airflow’s change detection is largely schedule-driven, and Prefect’s flow graph is dynamic at runtime with no built-in stale-asset model.
Dagster isn’t a free win. Two trade-offs come with it. Airflow has the deepest operational maturity and the largest connector ecosystem of the three — a decade of production hardening and integrations I’d have to build or wire up myself in Dagster. Prefect is friendlier for engineers who think in plain Python rather than in asset graphs. Neither of those is enough to offset the structural fit Dagster has for this pipeline, but they’re worth naming.
With the orchestrator chosen, the next post walks through the architecture of the NuGet classification pipeline in Dagster: the asset graph for the four phases, the partition scheme, where sensors and quality checks attach, and how the index phase exposes classifications to downstream consumers.