#043: 3 Must-Know Trends for Engineers | DataOps
May 24, 2023
DataOps is the name for a way of working where data is treated like a software product: built, tested, released and maintained continuously, often by more than one team. Three trends define it. Automation, where testing, deployment, infrastructure and scheduling run from code and a few simple commands. Integration, where the job becomes connecting many single purpose tools through triggers and alerting so they work together. And collaboration, where ownership of a pipeline spreads across teams, stakeholders get more involved, and documentation and lineage become a shared responsibility rather than an afterthought.
Key takeaways
- The lines around a data team are blurring. Modern data engineering is continuous and collaborative, and data is treated like a product the way a software application is.
- Automation starts with code based tools and version control. Once your project is code, testing and deployment can run through GitHub Actions or GitLab pipelines instead of a rigid release schedule.
- Infrastructure is code too. Terraform lets you define your entire Snowflake setup in files and deploy it automatically, with no manual changes in the console.
- The modern stack is many focused tools instead of one that does everything. They don't talk to each other unless you set up the triggers, so that becomes a big part of the job.
- Alerting has to be integrated, not emailed. Wire your scheduler, such as Airflow, into Slack so results reach the team without anyone checking manually.
- Expect to collaborate with reporting, automation and modeling teams, and with stakeholders who are more data literate and want a say in decisions.
- Treat data dictionaries and lineage as part of development. dbt builds some of this in, but a wiki or shared site works too. What matters is that it isn't an afterthought.
Why a data team looks different now
Ask what a data team is and most people picture a single group of developers with some QA or IT support around them. Today those lines are blurred.
Modern data engineering has become a continuous, collaborative effort that spans multiple teams. Data is viewed more like a product, the way you'd think about a traditional software application, and the approaches for building and releasing data products have evolved to match.
DataOps is the term that gets used for that workflow. The three trends below are the biggest ones inside it.
Trend 1: Automation
This one should come as no surprise. Rather than writing ad hoc queries and building reports, engineers are now expected to develop inside an environment surrounded by automation and continuous activity.
Testing and deployments
The biggest impact is on testing and deployment. The shift toward code based tools and version control makes development more fun, in my opinion, and it's what makes automated testing and deployment possible through a few simple commands.
GitHub Actions and GitLab pipelines are where you'll see this in practice. You build automated workflows tied directly to your project code, so development, testing and deployment become continuous instead of a rigid deployment schedule with ad hoc testing.
Infrastructure as code
Your infrastructure can be automated the same way. Infrastructure as code means defining databases, warehouses, roles and environments in files instead of clicking through a console.
Terraform is the open source tool I use for this. You can have your entire Snowflake setup written in code and deployed through Terraform, then edit it and spin up new environments however you need. The obvious benefit is that nothing gets changed manually in Snowflake itself.
Containerization
Containers are where names like Docker and Kubernetes come in. A container lets you quickly recreate a very specific environment setup, and it can plug into the automated workflows above.
You can get creative with how containers fit into testing and deployment. They make it possible to automate things that previously would have been unheard of.
Scheduling
Scheduling has been around forever, but it is becoming far more advanced. It used to be a feature inside an existing tool, such as SQL Server Agent. Now there are entire tools dedicated to it:
- Airflow
- Luigi
- Jenkins
- Your own Python script that kicks off tools in order
Again, most of this is possible because of the move to code based tools, where anything can be initiated by a basic command.
Trend 2: Integration
So many tools are cloud based now and designed for connectivity. A large part of the engineering job has become making sure the different platforms in the stack actually work together.
Triggers
The trend has been toward multiple individual tools for specific purposes rather than one major tool that does it all. Each piece is better, but they don't talk to each other unless you tell them to.
I think of it like an engineer in a manufacturing facility. That engineer makes sure every machine runs together in the right order. We do the same with our digital machines.
Expect a big part of the job to be setting up triggers inside a workflow and between tools so everything runs smoothly.
Alerting
All that automation means a lot more happens behind the scenes, and you can't sit and watch it. Whether it's a job that succeeded or an unexpected error, you need immediate alerting with the information integrated across tools.
A simple example is connecting a scheduling tool like Airflow to a team communication tool like Slack. Airflow results post to Slack automatically, so nobody checks manually or waits on an automated email.
Many tools, one stack
The modern data stack has gone from one or two primary tools for every function to a collection of many tools, each with a specific focus.
It can feel overwhelming at first. But this is where we get to actually build things, and seeing an integration work the way you intended is one of the more rewarding parts of the job.
Trend 3: Collaboration
As pipelines grow in scope and complexity, ownership of the different parts spreads across teams. It no longer stays within one data team.
Working across teams
In large companies it's common to have one team for reporting, one for automation and a separate one for data modeling. Working in that environment means collaborating with many more people to reach a goal.
That's an adjustment if you are used to owning everything inside a small team and never relying on an outside group. Common examples of what you'll coordinate:
- Development dependencies between teams
- Release schedules
- Letting other teams know about issues found during your development
More direct feedback from stakeholders
Business stakeholders are far more data literate than they used to be. They want to be involved in decisions and make suggestions.
Stakeholders are also likely to be other development teams now, because they're directly affected by what you build. Be prepared for a collaborative, fluid environment instead of hiding behind the mysterious black box of development.
Data dictionaries and lineage
People have always asked for logic definitions. A data dictionary is the catalog of those definitions, and data lineage is the record of where each piece of data came from and what it flows into. The growing complexity of pipelines makes both a group effort.
Getting it right takes collaboration between teams to keep definitions and lineage accurate. Tools like dbt have some of this built in, but there are other ways:
- A shared document site
- A custom hosted website
- An internal wiki
- Tools built specifically for cataloging and lineage
Treat documentation as part of the development process. Not an afterthought, not a nice to have. It's a deliverable alongside the code.
Key terms
DataOps
A way of working that applies software practices such as automation, continuous delivery and cross team collaboration to building and releasing data products.
Infrastructure as code
Defining databases, warehouses, roles and environments in version controlled files, deployed with a tool like Terraform, instead of configuring them by hand.
Containerization
Packaging an application and its exact environment into a container, using tools like Docker, so it can be recreated reliably anywhere, including inside automated workflows.
Orchestration
Scheduling and sequencing the steps of a pipeline across tools, handled by dedicated schedulers such as Airflow rather than a feature inside a single database.
Data lineage
The record of where each table or column originates and what depends on it downstream, maintained alongside the data dictionary so definitions stay accurate.
Common questions
What is DataOps in simple terms?
DataOps is treating data like a software product. Changes are version controlled, tested and deployed automatically, tools are integrated so they run and alert without manual effort, and ownership and documentation are shared across teams rather than held by one group.
What is the difference between DataOps and DevOps?
DevOps applies these practices to software applications. DataOps applies the same ideas, automation, integration and collaboration, to data pipelines and the products built from them. The tools overlap heavily: GitHub Actions, Terraform and Docker show up in both.
Do I need Terraform for a small data team?
Not on day one, but it pays off sooner than people expect. Once your Snowflake roles, warehouses and schemas live in code, you can recreate an environment or review a change the same way you review a dbt model. Manual console changes are where drift and surprises come from.
How should a data pipeline send alerts?
Push results into the channel the team already watches. The common pattern is a scheduler like Airflow posting job outcomes into Slack, so a failure is seen within minutes instead of being buried in an automated email.
Where should data documentation live?
Wherever the team will actually maintain it. dbt generates docs and lineage from the project itself, which keeps them close to the code. A wiki or shared site works too. The requirement is that updating documentation is part of finishing a change, not a separate task.
Related reading
- Data Automation (CI/CD) with a Real Life Example
- Modern Data Workflows, Explained
- Why Data Teams Need Version Control
- Don't Overlook Workflow (as a Small Data Team)
Final takeaway
Every data team I work with eventually runs into all three of these, usually in this order: first the manual deployments hurt, then the tools stop talking to each other, then someone outside the team needs to understand what was built. Treat automation, integration and collaboration as part of the job from the start and each one gets easier.
Additional Free Resources
Starter Guides & Checklists
Explore additional free resources built on the same patterns I use with real clients so you can build your own with structure and confidence. Topics include data architecture, modeling and more specifically for small data teams.
