An introduction to Data Warehouse main concepts — A Data Warehouse is a component where your data is centralized, organized, and structured according to your organization's needs. It is used for data analysis and BI processes. Data warehouses are not a new concept. In fact, the concept was developed in the late 1980s. But, it evolved over time. …

An approach towards simplifying data warehousing — Data Warehousing theory has a lot of useful resources. However, such a large amount of resources sometimes can be overwhelming — or at least it was for us when we started to build our data warehouse. Originally, when we started shaping our data warehouse architecture, the first question we asked…

Five processes to improve your data pipeline operability and performance — Building a data warehouse pipeline can be complex sometimes. If you are starting in this world, you will soon realize there is no right or wrong way to do it. It always depends on your needs. Yet, there a couple of basic processes you should put in place when building…

An implementation using AWS S3, AWS Glue, DBT, and AWS Redshift — In past posts, I’ve been talking about Data Warehouses, their basic architecture, and some basic principles that can help you to build one. Today, I want to show you an implementation of Data Warehouse on AWS based on a case study performed a couple of months ago. This implementation uses…

Data load processes should not be written twice, they should be generalized — We use Airflow as our data pipeline orchestrator in order to easily orchestrating and monitoring data processes. Particularly, we’ve been working on data load processes in order to make them easier. These processes allow us for extracting data from different sources and integrating them in our data lake—in this case…

An approach based on pure tasks — We integrate several data sources in a project comprising more than forty applications. Such sources are integrated by performing batch processes allowing for incrementally loading data from on-premise databases to AWS S3. We use Airflow as our data pipeline orchestrator in order to easily orchestrating and monitoring such processes. In…