The Starter Guide for Modern Data

Working in data over the past decade has been a wild ride. Yet despite the innovation, most folks I talk to are more overwhelmed than ever.

The good news is there's never been a more exciting time to work in data engineering. The downside is the constant flood of information is leading to indecision, insecurity and, oftentimes, a mess.

But it doesn't have to be this way.

Let's begin.

Problem 1Overwhelm

There are too many options.

▶ Watch · 3:53 The 3 Pillars of Data Engineering

TipFocus on components, not tools

Whether architecting or learning, most of the focus tends to start with tooling. But this leads to generic questions like:

• Which tools should I be using or learning about?
• Is [this tool] better than [that tool]?
• Etc.

This inevitably leads to many versions of the same image: a huge list of tools in the current market. And a stressful decision for you.

But if that's what you're looking for, here's my version:

But is this really helpful? To me, the answer is not really.

If we forget all of these logos, here are 5 questions to start with instead:

1. Where will we consolidate our data?
2. How will it get there?
3. How will we clean it up?
4. How will we analyze it?
5. How can we automate our workflow?

Until you have a solid process for each, everything else is a "nice to have". Or more often, just a distraction.

First, let's translate these questions into real architecture components:

1. Where will we consolidate our data?→Storage/Database
2. How will it get there?→Ingestion
3. How will we clean it up?→Transformation
4. How will we analyze it?→Reporting
5. How can we automate the whole process?→Version Control & CI/CD

Put these components together and you can form the foundations of your data architecture.

Here's a high-level view of The 3 Pillars of Data Engineering, with the 5 components listed at the bottom. (I like to think about moving from left to right.)

To some, this sounds overly basic. But many companies I speak to are far from this.

Skipping past this is like trying to build a new house but starting with fancy kitchen appliances. They don't matter if you don't first have a roof, plumbing and electrical in place.

Rather than worrying about the entire data landscape, start with the 5 components. For example:

1. Storage/Database: Snowflake, BigQuery, etc.
2. Ingestion: Airbyte, Fivetran, Stitch, etc.
3. Transformation: dbt
4. Reporting: Tableau, Power BI, Looker, etc.
5. Version Control: GitHub, GitLab, etc.

Note: The tools listed are some common ones, but by no means the only options!

The "right" tool will ultimately be based on a unique combination of:

• Budget
• Skillset
• Sources
• Preferences (self-host vs cloud, build vs buy, etc.)

The main point is you don't need "all of the things" just because you hear about them. Your tooling may also change over time (and that's okay).

Now you know how to identify your initial tools by focusing on 5 foundational components. But this is just one part of a bigger puzzle.

The next question becomes: how do they all work together in a way that makes sense?

Problem 2Structure

The end-to-end design is unclear.

▶ Watch · 9:51 The Gift & Curse of Small Data Teams

TipStart with a Simple Stack

There's no universal "best" strategy, but there are tried and true fundamentals.

I also believe that the key to a successful architecture isn't how advanced or complex it is. Rather, it's simplicity.

But, again, how do we do this?

I follow what I call "The Simple Stack", which can be defined as:

At a high level, it looks like this:

Let's break this down. Key attributes and ways to think about this design:

1. Cloud database: Take advantage of enhanced storage and performance capabilities.
2. Batch processing: Real-time sounds nice, but will it really change business decisions? Probably not.
3. 3-layer data pipeline
   • Staging: Clean and filter source tables up front (modularity)
   • Warehouse: Establish a well-structured data model (ex. facts and dimensions)
   • Marts: Present clean, user-friendly tables intended for analytics (ex. reporting)
4. Isolated environments
   • DEV: A safe space for each developer to build new changes
   • CI: A pre-production area for workflow automation and testing
   • PROD: The "live" datasets shared with stakeholders
5. Version control with automation: Track logic changes and automate data quality checks.

Again, the theme here is simplicity. When the time comes that you've outgrown these foundations, then consider introducing something new.

Just know that it's easy to onboard a new tool or process, but extremely difficult to undo it. So be patient.

Speaking of patience, the next challenge is to put these nice-looking designs into action. But this feels incredibly daunting, not to mention it takes forever... right?

Problem 3Time

This takes a long time to implement.

▶ Watch · 12:24 How to Build Modern Data Architectures (faster & with more structure)

TipReverse engineer a single deliverable

Once you have your stack outlined, it's time to implement. Teams I've been on in the past often followed something like this:

1. List all known data sources and start loading them all. Right away.
2. Divide and conquer, building every data model as fast as possible.
3. Share the "number of tables built" as a sign of productivity.

This reminds me of a quote from legendary college basketball coach John Wooden:

"Don't mistake activity with achievement."

We spend all this time planning, just to be in a rush to muddy it up trying to look busy.

Rather than trying to do it all, I suggest working with stakeholders to pick ONE key deliverable. For example, a specific report, group of metrics or data feed.

Then, use it as the guide to narrow the scope and build a structured end-to-end pipeline, one step at a time.

In my experience, the benefits of this approach include:

• Establishing a rock-solid foundation: You give yourself the headspace to address edge cases up front.
• Avoiding scope creep: You set clear expectations (ex. a single deliverable).
• Building trust with stakeholders: You get them a quick win.
• Generating momentum: You now have a repeatable, more efficient process going forward.

The exact steps may vary between teams and organizations. But hopefully this approach helps make this overwhelming task feel much more doable.

Common questions

What are the core components of a modern data architecture?

Five components: storage (a cloud database), ingestion, transformation, reporting, and version control with CI/CD. Until you have a solid process for each, everything else is a nice to have.

Which tools should I use for my data architecture?

It depends on your budget, skillset, sources and preferences. Common choices are Snowflake or BigQuery for storage, Airbyte, Fivetran or Stitch for ingestion, dbt for transformation, Tableau, Power BI or Looker for reporting, and GitHub or GitLab for version control.

What is the Simple Stack?

A lean data architecture designed for modern workflows, but with minimal tooling: a cloud database, batch processing, a 3-layer pipeline (staging, warehouse, marts), isolated DEV, CI and PROD environments, and version control with automation.

Do I need real-time data?

Probably not. Real-time sounds nice, but ask whether it will really change business decisions. For most teams, batch processing is enough.

How do I implement a data architecture faster?

Pick one key deliverable with your stakeholders, like a specific report, and build a structured end-to-end pipeline for it one step at a time. Then repeat that process for everything after.

Recap

▶ Watch · 15:03 What Every Data Team Lead Should Know (Lessons From 30+ Teams)

In this guide, we've addressed 3 of the biggest challenges with modern data engineering:

1. Overwhelm→Focus on the 5 key components (vs all tools)
2. Structure→Build a Simple Stack (vs complex)
3. Time→Focus on a single deliverable (vs everything at once)

If you've made it this far, you now have a plan of attack for each.

What I shared in this guide is exactly what I do to help many of my consulting clients. I hope it's also helped you feel more prepared to take it on in your own way.

Cheers,
Michael