The Starter Guide for Modern Data

Read time: 5 minutes

Working in data over the past decade has been a wild ride. Yet despite the innovation, most folks I talk to are more overwhelmed than ever.

The good news is there's never been a more exciting time to work in data engineering. The downside is the constant flood of information is leading to indecision, insecurity and, oftentimes, a mess.

But it doesn't have to be this way.

Let's begin.

Problem 1Overwhelm

There are too many options.

TipFocus on components, not tools

Whether architecting or learning, most of the focus tends to start with tooling. But this leads to generic questions like:

• Which tools should I be using or learning about?
• Is [this tool] better than [that tool]?
• Etc.

This inevitably leads to many versions of the same image: a huge list of tools in the current market. And a stressful decision for you.

But if that's what you're looking for, here's my version:

But is this really helpful? To me, the answer is not really.

If we forget all of these logos, here are 5 questions to start with instead:

1. Where will we consolidate our data?
2. How will it get there?
3. How will we clean it up?
4. How will we analyze it?
5. How can we automate our workflow?

Until you have a solid process for each, everything else is a "nice to have". Or more often, just a distraction.

First, let's translate these questions into real architecture components:

1. Where will we consolidate our data?→Storage/Database
2. How will it get there?→Ingestion
3. How will we clean it up?→Transformation
4. How will we analyze it?→Reporting
5. How can we automate the whole process?→Version Control & CI/CD

Put these components together and you can form the foundations of your data stack.

Here's a high-level view of The 3 Pillars of Data Engineering, with the 5 components listed at the bottom. (I like to think about moving from left to right.)

To some, this sounds overly basic. But many companies I speak to are far from this.

Skipping past this is like trying to build a new house but starting with fancy kitchen appliances. They don't matter if you don't first have a roof, plumbing and electrical in place.

Rather than worrying about the entire data landscape, start with the 5 components. For example:

1. Storage/Database: Snowflake, BigQuery, etc.
2. Ingestion: Airbyte, Fivetran, Stitch, etc.
3. Transformation: dbt
4. Reporting: Tableau, Power BI, Looker, etc.
5. Version Control: GitHub, GitLab, etc.

Note: The tools listed are some common ones, but by no means the only options!

The "right" tool will ultimately be based on a unique combination of:

• Budget
• Skillset
• Sources
• Preferences (self-host vs cloud, build vs buy, etc.)

The main point is you don't need "all of the things" just because you hear about them. Your tooling may also change over time (and that's okay).

Now you know how to identify your initial tools by focusing on 5 foundational components. But this is just one part of a bigger puzzle.

The next question becomes: how do they all work together in a way that makes sense?

Problem 2Structure

The end-to-end design is unclear.

TipStart with a Simple Stack

There's no universal "best" strategy, but there are tried and true fundamentals.

I also believe that the key to a successful architecture isn't how advanced or complex it is. Rather, it's simplicity.

But, again, how do we do this?

I follow what I call "The Simple Stack", which can be defined as:

At a high level, it looks like this:

Let's break this down. Key attributes and ways to think about this design:

1. Cloud database: Take advantage of enhanced storage and performance capabilities.
2. Batch processing: Real-time sounds nice, but will it really change business decisions? Probably not.
3. 3-layer data pipeline
   • Staging: Clean and filter source tables up front (modularity)
   • Warehouse: Establish a well-structured data model (ex. facts and dimensions)
   • Marts: Present clean, user-friendly tables intended for analytics (ex. reporting)
4. Isolated environments
   • DEV: A safe space for each developer to build new changes
   • CI: A pre-production area for workflow automation and testing
   • PROD: The "live" datasets shared with stakeholders
5. Version control with automation: Track logic changes and automate data quality checks.

Again, the theme here is simplicity. When the time comes that you've outgrown these foundations, then consider introducing something new.

Just know that it's easy to onboard a new tool or process, but extremely difficult to undo it. So be patient.

Speaking of patience, the next challenge is to put these nice-looking designs into action. But this feels incredibly daunting, not to mention it takes forever... right?

Problem 3Time

This takes a long time to implement.

TipReverse engineer a single deliverable

Once you have your stack outlined, it's time to implement. Teams I've been on in the past often followed something like this:

1. List all known data sources and start loading them all. Right away.
2. Divide and conquer, building every data model as fast as possible.
3. Share the "number of tables built" as a sign of productivity.

This reminds me of a quote from legendary college basketball coach John Wooden:

"Don't mistake activity with achievement."

We spend all this time planning, just to be in a rush to muddy it up trying to look busy.

Rather than trying to do it all, I suggest working with stakeholders to pick ONE key deliverable. For example, a specific report, group of metrics or data feed.

Then, use it as the guide to narrow the scope and build a structured end-to-end pipeline, one step at a time.

In my experience, the benefits of this approach include:

• Establishing a rock-solid foundation: You give yourself the headspace to address edge cases up front.
• Avoiding scope creep: You set clear expectations (ex. a single deliverable).
• Building trust with stakeholders: You get them a quick win.
• Generating momentum: You now have a repeatable, more efficient process going forward.

The exact steps may vary between teams and organizations. But hopefully this approach helps make this overwhelming task feel much more doable.

Recap

In this guide, we've addressed 3 of the biggest challenges with modern data engineering:

1. Overwhelm→Focus on the 5 key components (vs all tools)
2. Structure→Build a Simple Stack (vs complex)
3. Time→Focus on a single deliverable (vs everything at once)

If you've made it this far, you now have a plan of attack for each.

What I shared in this guide is exactly what I do to help many of my consulting clients. I hope it's also helped you feel more prepared to take it on in your own way.

Cheers,
Michael