Get in Touch

#017: The 3 Pillars of Data Engineering

Oct 29, 2022

Since 2016 I’ve worked on 7 data platforms in all sizes.

Nowadays, I can grasp an architecture within a few conversations.

But in the early days I often felt lost and overwhelmed.

Today, I’ll share the 3 pillars I focus on to quickly understand any data architecture:

  1. Sources
  2. Data warehouse
  3. Insights


Source systems reflect the business

No company operates using only one tool.

And an architecture isn’t effective if key data points are missing.

Ultimately, everything starts from source integrations.

Which is why the first pillar is identifying sources and how data gets extracted.


Example: 20 sources synced via Fivetran & custom Python scripts.


The data warehouse is the hub

Accessing data is one thing, organizing it is another.

It can be exhausting hearing about tool selection and approaches.

Instead, find the data warehouse and notice how everything moves around it.

As an engineer, the second pillar is honing in on this central hub.


Example: Snowflake is the Airbyte destination, dbt environment and Tableau data source.


Insights are the goal

Businesses don’t invest in data just for fun.

It adds complexity that has real costs.

The value comes from insights and improved decision-making.

Learn how insights are presented and you’ve uncovered the third pillar.


Example: Power BI dashboards accessed by stakeholders throughout the business.



99% of platforms revolve around sources, a warehouse and insights.

Focus on these pillars and you’ll quickly see the big picture.

Level-up your abilities as a Data Engineer, faster.

Learn new data engineering tips, tricks and best practices every Wednesday.

Other Recent Posts

Data Automation (CI/CD) with a Real Life Example

May 17, 2023

3 Ways to Deploy Data Projects

May 10, 2023

The Importance of Virtual Environments

Apr 26, 2023

How to Create a Virtual Machine on GCP

Apr 19, 2023