#72: Managing the "End" of a Data Pipeline
Jul 10, 2024If there is such thing as the "end" of a data pipeline, it's typically a report.
Or something getting into the hands of our stakeholders.
But if you've ever built one of these reports you know that it's never just a simple handoff.
There's almost always some sort of back and forth.
Or future requests for adjustments.
So what I want to talk about in this video is about where you decide to actually make those changes.
And the secondary impacts of that decision.
The two common schools of thought I've seen are:
- Making changes directly inside the report
- Keeping changes in the database (transformation layer code)
In this video, I want to make a case for why I personally think you're better off going with option 2.
But whether or not you agree with me...
Hopefully it'll encourage you to consider what's best for your team (& company) going forward.
The main things we'll cover include:
- The tradeoffs between storing logic in reports vs database code
- The impacts on logic visibility & consistency
- Controlling the human/management element
Enjoy!
Michael
When you're ready, here are 2 other ways I can help you:
1. Simple Stack Academy: Level-up your data career by learning more than just SQL. Design, build & automate an end-to-end data architecture using modern tools like dbt⢠& Github. (Free 7 Day Preview)
2. Consulting Services: Get personalized support to properly implement a well-structured, scalable, and maintainable data architecture at your company.