The Modern Data Viewpoint (3 Principles)

The modern data viewpoint comes down to three principles: analytics is collaborative, analytics code is an asset, and analytics requires automated tools. It was championed by the people at dbt Labs, and it describes a posture toward the work rather than a tool you install. Collaborative means version control, quality assurance, documentation and modularity. Code as an asset means environments, service level guarantees and designing for maintainability. Automated tools means you should be able to run the whole pipeline with a single command instead of doing things by hand that software engineers stopped doing years ago. Building a good architecture depends at least as much on this perspective as on which tools you pick.

Where the viewpoint comes from

Building a solid data architecture is more than the tools you pick or the skills you have. A lot of it comes down to your overall perspective on analytics as a whole.

That idea of a viewpoint has been championed particularly well by the people at dbt Labs. I refer to it as the dbt viewpoint, though nothing here is affiliated with them. This is my interpretation of what they published.

What follows was originally a lesson I recorded for a dbt course a few years ago. The ideas have aged better than most things in this field.

Analytics teams have a workflow problem

The quote I start with is from Tristan Handy, the CEO and founder of dbt Labs: analytics teams have a workflow problem.

That single line summarizes the whole motivation. Data has long been its own little world inside companies, with its own way of doing things and no formal workflow.

The gap with software engineering

Software engineering became modern, automated and fluid. Data stayed rigid. You build something, deploy it, make a report, repeat.

Bridging that gap, and specifically how teams work together while doing it, is a large part of what the viewpoint is about.

A short history of data engineering

It helps to see where the current approach came from, because none of it was obvious at the time.

Early 2000s

Traditional on premise databases, managed almost exclusively by DBAs, short for database administrators. The idea of a data team is not really a thing at most companies yet.

Mid to late 2000s

Nearly every company has a data warehouse or is building one. ETL pipelines bring a more structured way of moving data, and the volume keeps growing.

By 2008 and 2009 the social media companies are reshaping both how data looks and how we interact with it as a society.

Early to mid 2010s

Cloud computing arrives in force with Azure and Google Cloud. At the same time, data teams get more siloed.

People across the company become more data literate, so instead of one centralized team you get one off reports and queries everywhere. Pipelines shift from ETL toward ELT because the volume no longer allows the old approach.

Early 2020s and beyond

Analytics engineering emerges as a discipline, along with DataOps, a more automated and collaborative way of operating than the siloed model before it.

Tooling specializes. One product used to extract, transform and load. Now there is dbt for transformation, Fivetran, Airbyte and Stitch for extract and load, and Tableau or Power BI for reporting.

A few founding dates put it in perspective: SSIS as Microsoft's ETL tool, GitHub in 2008, Snowflake in 2012, dbt in 2016.

Row based versus columnar databases

This distinction matters because it shapes how dbt works under the hood and why some practices make sense at all.

Row based is the traditional kind: SQL Server, Postgres, MySQL. A query reads one row at a time, which suits CRUD operations, meaning create, read, update and delete.

What columnar changes

A columnar database reads by column instead. On very wide tables you only touch the columns you asked for, which is far more efficient for analytics at scale.

The trade is that it is poor at individual transactional writes and updates. Different purpose, different design.

Analytics went columnar, and that is the ground the modern stack is built on. Snowflake, Databricks, BigQuery, Synapse and Redshift are the familiar examples.

If the distinction is new to you, it is worth reading further. Knowing which type you are working on explains a lot about why certain patterns get recommended and others do not.

The three principles

The viewpoint itself summarizes into three statements. Each one unpacks into a handful of practices.

Analytics is collaborative

• Version control. The baseline for working together on anything.
• Quality assurance. Treat the data as something worth taking care of rather than something that just has to land.
• Documentation. Written down, not held in somebody's head.
• Modularity. Package logic into blocks so a fix in one place flows everywhere, instead of being chased across many copies.

A lot of this sounds obvious now. Set against the history above, almost none of it was standard practice.

Analytics code is an asset

• Environments. Development, a QA or staging layer, and production. Not straight to production.
• Service level guarantees. Commitments about what the data does and when it arrives.
• Designed for maintainability. Built to be changed later, the way a software product is.

How code moves between those environments is as much a part of the asset as the code itself.

Analytics requires automated tools

We should not be doing manual work that software engineers stopped doing long ago. The standard worth aiming at is running the entire pipeline with a single command.

If a step only happens because somebody remembered to do it, that is the gap the principle is pointing at.

In practice that means scheduled runs, automated tests and a deployment you trigger rather than perform by hand. The goal is not fewer people, it is fewer steps that exist only because nobody automated them.

Viewpoint plus tool

When I think about dbt as a whole, it is the viewpoint combined with the tool, whether that is the open source version or dbt Cloud.

You need both halves. The tool without the perspective becomes a nicer way to write the same tangle. The perspective without a tool stays an aspiration.

Together they give you an approach you can actually build transformations on, which is why the ideas outlast any particular release.

Related reading

• Don't Overlook Workflow (as a Small Data Team)
• Avoid These 5 Costly Mistakes (as a small data team)
• 3 Strategy Killers for Data Teams
• Can Small Data Teams Afford a "Modern" Database?

Final takeaway

When I review an architecture with a client, the gap is rarely the tool. It is that nobody ever decided what the team believes about how analytics should be built. Settle the viewpoint first and most of the tooling arguments answer themselves.