Thoughts on software engineering, data, and the things I'm building and learning along the way.
Deploying the pipeline against the full NuGet catalog, reading what the data actually says, and the AND/OR classifier bug the unit tests were perfectly happy with.
Read MoreThe wiring: how the architecture from Part 3 becomes actual Dagster assets, Postgres tables, and the pure classification function at the center of it.
Read MoreTurning the requirements into an architecture: the asset graph, Postgres schema, schedules, quality gates, and freshness policies for the NuGet classification pipeline.
Read MoreComparing Dagster, Apache Airflow, and Prefect against six requirements to pick an orchestrator for a NuGet package classification pipeline.
Read MoreSix requirements I'd hold any data orchestrator to before building a NuGet package classification pipeline.
Read More