Data Pipeline Builder

Casos de uso

• Generating Airflow DAGs for complex ETL workflows
• Creating dbt models with proper staging, intermediate, and mart layers
• Designing Kafka streaming pipelines with schema registry
• Building data quality validation rules with Great Expectations
• Setting up incremental data loading patterns

Design a data pipeline to ingest e-commerce order data.

Source: PostgreSQL (orders, order_items, customers, products tables)
Destination: Snowflake data warehouse
Schedule: Every 15 minutes (near real-time)

Requirements:
1. Incremental extraction using CDC (Change Data Capture)
2. dbt transformation layer with:
   - Staging models (1:1 source mapping)
   - Intermediate models (joins, deduplication)
   - Mart models (fact_orders, dim_customers, dim_products)
3. Data quality checks after each layer
4. Schema evolution handling (new columns, type changes)
5. Alerting on pipeline failures or quality issues

Generate:
- Airflow DAG for orchestration
- dbt models (SQL + schema.yml)
- Data quality assertions
- Monitoring dashboard queries

Modelos recomendados

Herramientas compatibles

Modalidades

Skills relacionadas