#78: How Data Teams Maintain Production (with high quality)

Data teams maintain production quality with a few quiet habits that catch problems before a stakeholder does. A linter checks every query and file against a shared set of rules, so a missing comma or a misnamed file never reaches code review. Separate development, CI and production environments give every change at least two places to fail safely before it goes live. Automation deploys and tests changes in CI whenever a pull request opens, and keeps production refreshed on a schedule once they merge. A pull request template turns the review into a checklist instead of a formality. None of these are dramatic on their own, and together they are what earns the business's confidence in data it cannot see behind.

Why this comes down to confidence

How a team deploys changes to production is one of its most critical processes. Go a level deeper and the reason it matters is confidence.

Part of that is your own confidence, and your team's, that you are building something valuable. The bigger part is the confidence your stakeholders and the business have in the data you hand them.

To everyone outside the team, the pipeline is a black box. They can't see how a number was produced. They are relying on you to have a process that makes sure it is right before it reaches them.

Below are three ways to improve your testing before changes reach production, plus a bonus. My hope is that you find one or two you can fold into your own process.

1. Add a linter

What a linter does

A linter is a small program you install that checks your queries and files against a set of rules. It exists to maintain consistency, and it catches the little things that are easy to overlook:

• A missing comma at the end of a statement
• File names that don't follow the convention
• Improper references between objects
• Spelling mistakes

You can run it as a command while you develop, or automate it so it runs as part of pre-production testing. Linters exist for SQL, Python and most other languages, so look one up for whatever your team writes.

The team dynamics benefit

The less obvious value is in code reviews. Nobody likes being the person who points out small errors, and nobody enjoys receiving that feedback either. It feels nitpicky from both sides.

With a linter in play, the program calls it out. Recommendations are a lot easier to accept from a tool than from someone on the team, and the human review can focus on the logic.

2. Isolate your work in environments

When teams are first getting started, and often well past that point, they tend to have one environment: production. Everything else is loosely called development, but there is no process behind it.

Being deliberate about environments gives you different spots to catch errors before production catches you off guard. Here is how I like to set it up.

Development

Each developer gets their own development schema. Everything they build deploys there.

Don't break it into sub schemas. Production might have several schemas, but in development everything lands in the one schema for that person.

Two things come from this. You have a safe space to build, and you avoid conflicts with other developers, so you can trust the data you are looking at.

CI, test or UAT

The second environment is CI, short for continuous integration. Call it test or UAT if you prefer. It is the place to do a full deployment before production.

This happens during the merge process, as you move a change along. You can run tests and do data validation here. Unlike development it is a shared environment, and it is a second chance to confirm everything works, happening behind the scenes.

Production

Once everything looks good, you merge to production. This is where your BI tools connect and where stakeholders get their data, and it can be locked down accordingly.

You might break production into separate schemas and keep the lower environments consolidated. That is up to your setup. The point is that by the time anything reaches production it has already passed through two other environments.

3. Automate the path between them

Automation is what ties the environments together. A lot of data engineering is building a system, and a system is a set of individual parts that only becomes valuable once they are connected.

Deploy to CI on every pull request

Start with the move from development to CI. Set up automation that deploys and tests your changes into the CI environment whenever you open a pull request or merge request in your version control platform.

You can build these pipelines with whatever steps, checks and commands you want to run. The point is that the deployment and the testing happen automatically, instead of always relying on a person to check everything perfectly.

Refresh production on a schedule

The other half of automation is after the merge. Once a change is in production you want it kept updated and refreshed.

That can be a simple cron job or a separate orchestration tool. Either way the idea is the same: continuously run and test things even after they are live.

Bonus: a pull request template

One more piece of advice for anyone working with a version control platform. Create a request template, whether that is a pull request template on GitHub or a merge request template on GitLab. Nearly every platform supports it.

What to put in it

A template is a Markdown file that auto-populates the description of every new request. Mine usually include:

• An overview, a quick description of what changed and why
• A checklist of the specific items you should always double check before merging

Why it works

The value isn't that we want more documentation. Nobody likes writing it. The alternative is that people cut corners, because if there is an easier way to do something, that is the way it gets done.

A template is a forcing function. It reminds people what to look for instead of letting them push a change through quickly, skip the context, and have it come back as a stakeholder reporting a production issue.

It pays off later too. When something does break, a good description tells you what changed and what to fix. Without one you are blind, digging through the code to reconstruct what happened, and that isn't fun for anybody.

Putting it together

To recap, three ways to improve testing before anything reaches production:

• Add a linter so the rules are enforced by a program, not a reviewer
• Separate your work into environments so every change has two places to fail before production
• Use automation along with templates to double check your work and leave a record of it

Pairing the automation with the templates is the combination I see pay off most. One makes sure the checks run, the other makes sure people think.

Related reading

• The 3-Environment Design for Your Database (DEV vs CI vs PROD)
• dbt Environments vs Targets: What's the Difference?
• Data Automation (CI/CD) with a Real Life Example
• Why Data Teams Need Version Control

Final takeaway

None of these habits are dramatic, and that is why they get skipped. On the teams I work with, the ones stakeholders trust have a linter, three environments and an automated check on every pull request running quietly in the background. Put one in place and you might just get an unexpected thank you from a stakeholder.