#017: The 3 Pillars of Data Engineering

newsletter Oct 29, 2022

Since 2016 I’ve worked on 7 data platforms in all sizes.

Nowadays, I can grasp an architecture within a few conversations.

But in the early days I often felt lost and overwhelmed.

Today, I’ll share the 3 pillars I focus on to quickly understand any data architecture:

  1. Sources
  2. Data warehouse
  3. Insights

 

Source systems reflect the business

No company operates using only one tool.

And an architecture isn’t effective if key data points are missing.

Ultimately, everything starts from source integrations.

Which is why the first pillar is identifying sources and how data gets extracted.

 

Example: 20 sources synced via Fivetran & custom Python scripts.

 

The data warehouse is the hub

Accessing data is one thing, organizing it is another.

It can be exhausting hearing about tool selection and approaches.

Instead, find the data warehouse and notice how everything moves around it.

As an engineer, the second pillar is honing in on this central hub.

 

Example: Snowflake is the Airbyte destination, dbt environment and Tableau data source.

 

Insights are the goal

Businesses don’t invest in data just for fun.

It adds complexity that has real costs.

The value comes from insights and improved decision-making.

Learn how insights are presented and you’ve uncovered the third pillar.

 

Example: Power BI dashboards accessed by stakeholders throughout the business.

 

 

99% of platforms revolve around sources, a warehouse and insights.

Focus on these pillars and you’ll quickly see the big picture.

Level-up your abilities as a Data Engineer, faster.

Subscribe to receive tips to improve your skillset as a data engineer every Saturday. Always readable in 2 minutes or less.

Other Recent Posts

#021: Why your data team needs version control

Nov 26, 2022

#020: What to Learn First as a Data Engineer

Nov 19, 2022

#019: 3 Time-Saving dbt Cloud Features

Nov 12, 2022

#018: Lessons Going from Snowflake to BigQuery

Nov 05, 2022