Jordan Calloway

CLI tool for scaffolding production-ready Apache Airflow DAGs from YAML specs, with built-in retry logic, SLA alerting, and data quality checks via Great Expectations.

• —Built a Python CLI using Click that generates fully configured Airflow DAGs from a YAML spec — including task dependencies, retry policies, SLA alerting, and Slack notifications.
• —Integrated Great Expectations checkpoint generation so every scaffolded pipeline ships with schema validation and row-count anomaly detection out of the box.
• —Added a local dev mode using Docker Compose that spins up Airflow, PostgreSQL, and a mock data source in under 60 seconds for rapid iteration.

Reference architecture for real-time financial event processing using Kafka and Flink, with a live React dashboard. Built to explore sub-second streaming pipelines.

• —Built a Kafka producer simulating realistic financial transaction streams at 10k events/second, with configurable fraud patterns for testing downstream anomaly detection.
• —Implemented a Flink streaming job for real-time aggregations — rolling totals, per-merchant spend windows, and anomaly flagging — with sub-500ms end-to-end latency.
• —Wired a React dashboard with WebSocket updates to visualize live transaction volume, flagged events, and per-category breakdowns.

A collection of dbt macros for automated audit logging, freshness checks, and row-count reconciliation across Snowflake and BigQuery models.

• —Wrote a suite of dbt macros for automated audit logging — tracking row counts, null rates, and schema changes on every model run without custom Python.
• —Added freshness check macros compatible with both Snowflake and BigQuery that surface stale source tables before they silently corrupt downstream models.
• —Packaged as a dbt Hub-compatible package with full documentation and example project so teams can install and configure in under 10 minutes.

Python library for CDC-aware incremental data loading into Snowflake — supports PostgreSQL, MySQL, S3, and REST API sources with automatic schema evolution.

• —Built a Python library that detects changed rows using configurable watermark columns or CDC log parsing, and loads only deltas into Snowflake — reducing warehouse credit usage significantly.
• —Supports multiple source connectors out of the box: PostgreSQL, MySQL, S3 CSV/Parquet, and REST APIs with pagination handling.
• —Added schema evolution handling — automatically ALTERs target Snowflake tables when source columns are added or types change, with configurable safety thresholds.

Headless metrics store built on dbt and Snowflake — define business metrics once in YAML and query them via REST API or a lightweight React explorer. Eliminates duplicated metric logic across BI tools.

• —Built a YAML-based metric definition layer on top of dbt — teams define metrics once with dimensions, filters, and time grains, and MetricLayer generates the underlying SQL automatically.
• —Implemented a FastAPI query layer that translates metric + dimension + date range requests into optimized Snowflake SQL, with Redis caching for frequently queried combinations.
• —Built a lightweight React explorer UI for browsing defined metrics, running ad-hoc queries, and exporting results to CSV — usable by non-technical stakeholders without SQL knowledge.