The 10 Modern Data Components

The further you get into data engineering, the more you realize how many tools and components there are. That's exciting, but it also leads to a lot of complexity too early in a lot of architectures.

This guide breaks it down into the 10 key components I see work for the vast majority of teams. For each one, you'll see what it is, how it's used, why it helps and some example tools.

It's not about all the different tool options. What matters is understanding where each component fits in the bigger picture.

Part 1The essentials

The 5 components every data architecture needs.

▶ Watch · 13:42 The 10 Modern Data Components - Essentials (Part 1)

Component 1Storage & databases

A place to put your data.

At some point, you need to put data somewhere. Without it, there's really not much else you can do with your data.

Storage usually means file storage, like S3 buckets or blob storage. That's where you'll see a data lake, with raw data in all different formats. Databases are where most of us spend our time. I think of the database as the hub of all your information, where your data warehouse, marts and data modeling live.

Component 2Ingestion

How the data gets there.

We have a place to store data. But how does it get there? To me, that's a foundational, incredibly important component.

You can batch load on a schedule or stream data in real time. For most of the clients I work with, batch loading is really all you need. Either way, it's still ingestion: moving data from a source system into storage.

Component 3Transformation

Turning raw data into something useful.

Once the data is loaded, what do we do with it? Raw data by itself isn't that helpful most of the time. Transformation applies real-world rules and business logic, and maybe most importantly, gives your data organization and consistency.

This is where your data models live: a staging layer, a warehouse with facts and dimensions, and marts ready for reporting. The trend is toward code-based tools like dbt, because you can customize and automate a lot more. Drag-and-drop tools still have their place.

Component 4Reporting & analytics

Where the business puts data to use.

This is the part that gets all the glory. We do all this work behind the scenes, and this is what everybody sees. It's where the business gets insights, analyzes history, makes predictions and makes better decisions.

Component 5Version control & CI/CD

Track every change and automate the workflow.

I group these together because you get the most value using them together. Version control tracks every change in a structured, transparent way. CI/CD automates tasks like testing and deployments.

Each developer works in their own branch and development area, separate from production. Changes move to a pre-production environment (QA, UAT or test), trigger automated checks, and get reviewed before they reach production. You can even run your refresh schedule from the same platform before you need a dedicated orchestrator.

Part 2The advanced components

Add these once the essentials are in place.

▶ Watch · 12:45 The 10 Modern Data Components - Advanced (Part 2)

There are way more things you can add to an architecture. These five are the ones I see teams reach for next. Each one is worth knowing, but most teams don't need them on day one.

Component 6Orchestration

The control plane for your whole pipeline.

An orchestrator connects and triggers the different parts of your pipeline from a single tool. Without one, the process can get hard to manage, and you lose visibility into what's happening where.

This one is on the border of being an essential. I list it as advanced because you can handle a lot of orchestration early on with a simple scheduled job in your version control platform. Once ingestion, transformation and refreshes are all running at different times, an orchestrator is a great next step to put it all together.

Component 7Containers

Isolated environments without new machines.

A container is an isolated, virtual instance of an operating system that runs on top of your hardware. It gives you a clean, separate place to run applications and processes. Docker is by far the most common tool.

Some teams never use containers and are perfectly successful. Others rely on them heavily. Think of it as an infrastructure piece you can lean on when you need it.

Component 8Infrastructure as code (IaC)

Manage cloud resources with code, not clicks.

Instead of manually clicking through settings to create a database, network, virtual machine, users or permissions, you define it in code and deploy it.

The big benefit is transparency. In the past, one person with admin access might change permissions with no trail of what happened. With infrastructure in code, combined with version control and CI/CD, everyone can see exactly what's deployed. Like the other advanced components, add it if and only if you need it.

Component 9Data quality

Add-on tools for accuracy, lineage and trust.

Data quality itself isn't less important. If anything, it should be one of the most important things. This component is about the separate data quality and governance tools you add once the foundations are in place, since there are other ways to manage quality before this point.

Some of it is already built into your tools. dbt has tests, for example. AI is also making documentation and testing easier, because it can understand the context of your code.

Component 10Reverse ETL

Send clean data back to the business apps.

Reverse ETL sends data that's been through your pipeline back into your business applications. A simple example is populating custom fields in your CRM, like Salesforce, from your transformed data.

It lets business users stay in the tools they know while using the same business-approved logic as the data team. Otherwise they end up recreating the logic on their own, and now you have two versions of the same number. Most teams don't need this for a while, but it's a concept worth knowing.

All 10 at a glance

Common questions

What are the components of a modern data architecture?

The 10 key components are storage and databases, ingestion, transformation, reporting and analytics, version control and CI/CD, orchestration, containers, infrastructure as code, data quality and reverse ETL. The first five are the essentials.

Which components do I need to start?

The five essentials: storage and databases, ingestion, transformation, reporting and analytics, and version control with CI/CD. With those in place, you're in a great position to build and scale your architecture. Add the advanced components only when you need them.

Do I need an orchestration tool?

Not right away. A scheduled job in your version control platform can handle a lot at first. An orchestrator is worth it once several processes run at different times and you need one place to monitor and control them.

Do I need real-time streaming?

Most teams don't. For most of the companies I work with, batch loading on a schedule is all they need.

What's the difference between storage and a database?

Storage usually means file storage, like S3 or Azure Blob, where a data lake holds raw data in any format. A database adds structure, performance and security, and it's where your data warehouse and data models live. Some architectures skip file storage and load straight into a database.

What is reverse ETL?

Reverse ETL sends clean, transformed data from your warehouse back into business applications like a CRM, so business users can work in the tools they know with the same numbers as the data team.

Cheers,
Michael