#72: Managing the "End" of a Data Pipeline
Jul 10, 2024If there is such thing as the "end" of a data pipeline, it's typically a report.
Or something getting into the hands of our stakeholders.
But if you've ever built one of these reports you know that it's never just a simple handoff.
There's almost always some sort of back and forth.
Or future requests for adjustments.
So what I want to talk about in this video is about where you decide to actually make those changes.
And the secondary impacts of that decision.
The two common schools of thought I've seen are:
- Making changes directly inside the report
- Keeping changes in the database (transformation layer code)
In this video, I want to make a case for why I personally think you're better off going with option 2.
But whether or not you agree with me...
Hopefully it'll encourage you to consider what's best for your team (& company) going forward.
The main things we'll cover include:
- The tradeoffs between storing logic in reports vs database code
- The impacts on logic visibility & consistency
- Controlling the human/management element
Enjoy!
Michael
Looking for more? Here are 2 other ways I can help you:
1. Simple Stack Academy: Join 100+ students in an end-to-end, project-based training. Simple Stack Academy teaches you a 5-Step Framework to design, build & automate reliable end-to-end data architectures using modern tools like dbt & Github. Ideal for both data engineers looking to advance their careers and data team leaders looking for a clear strategy to follow.
2. Consulting Services: Get personalized support to properly implement a well-structured, scalable, and maintainable data architecture at your company. Whether starting from scratch or migrating a legacy design, you'll get custom support each step of the way to make sure it's done right.