Modern Data Stack Data Warehouse
As the lead data engineer and manager of the data & analytics department at OHB I was responsible for the end to end ideation, design and implementation of this enterprise data warehouse project. Keeping aligned with cutting edge industry trends I architected a solution inspired by the ‘modern data stack’ opting into Snowflake as the primary data store, Fivetran as the primary ETL service, and the Dbt framework for accurate and reproducible CI/CD data modelling. This stack was chosen to be highly scalable, cost effective and risk-averse considering the limited in-house developer resources at the time as the team was actively growing. The wider system included a data audit system complete with monitoring dashboards, a self-hosted and maintained orchestration platform (Apache Airflow), and ancillary data stores (AWS RDS) and data pipelines handling legacy data and fringe data sources that couldn’t be handled by Fivetran alone.
- PLATFORMBackend, DB
- STACKSnowflake, AWS S3, Glue, EMR, Fivetran, Dbt, Python, Airflow
- WEBSITEhttps://www.ozhairandbeauty.com
- GITHUB