You have a daily drop of 10,000 JSON log files on S3. You want to transform them and load into Postgres. Airflow means a scheduler, a metadata DB, a webserver, DAG files, and operators. Dagster and ...

Este projeto implementa um pipeline ETL que coleta dados meteorológicos de São Paulo a cada hora, processa as informações e armazena em um banco de dados PostgreSQL para análise posterior.

ETL, or Extract Transform Load, is essential for data preparation and migration across various platforms. Apache Airflow enables users to create automated workflows using Python features, suitable for ...